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EXECUTIVE  SUMMARY 


Civilian  trauma  surgical  experience  has  decreased  for  four  reasons:  1)  the  Accreditation 
Council  of  General  Medical  Education  (ACGME)  committee  2003  ruling  reduced 
training  hours  from  30,000  to  19,200;  2)  Endovascular  balloon  occlusion  and 
embolization  has  replaced  many  open  surgeries  for  emergency  control  of  hemorrhage;  3) 
Fewer  motor  vehicle  occupant  injuries  and  gunshot  wounds  nationwide;  4)  New  blood- 
use  protocols  (1:1:1  red  cell:  plasma:  platelet)  and  tranexamic  acid  reduced  the  need  for 
open  surgical  interventions  to  control  bleeding. 

PROBLEM:  Reduced  clinical  opportunities  for  open  surgical  control  of  hemorrhage 
and  lack  of  surgeon  technical  skills  performance  metrics  opened  a  large  capability 
gap  transcending  U.S.  military  and  civilian  need.  Furthermore,  the  absence  of 
validated  competency  metrics  impedes  assessment,  making  it  unclear  whether  these 
technical  skills  improve  with  training,  if  these  skills  degrade  with  time  since 
training,  or  if  there  is  a  need  for  refreshing  these  skills. 

Approach:  The  study  analyses  of  were  anchored  around  the  ASSET  course  training,  a 
one  day  cadaver-based  course  covering  59  different  trauma-related  procedures.  Since 
there  are  no  available  metrics  to  quantify  training  benefits  of  the  ASSET  course,  we 
developed,  tested,  and  validated  surgeon  performance  metrics  for  non-technical  and 
technical  skills  acquired  in  the  ASSET  course.  We  used  both  unpreserved  cadavers  and 
realistic  physical  models,  testing  surgeons  before  and  after  ASSET  training  and  up  to  4 
years  later  . 


Five  questions  were  addressed  by  this  study:  Do  core  trauma  surgical  procedural  skills 
show  improvement  with  training?  Which  components  among  these  skills  benefit  most 
from  training?  Does  training  reduce  the  occurrence  of  errors?  Can  we  identify  a  time 
since  training  when  skills  may  need  to  be  refreshed?  Can  a  hyper-realistic  physical 
model  replace  the  unpreserved  cadavers  used  in  this  study? 

We  enrolled  106  surgeons  recruited  from  13  different  residency  training  programs,  from 
surgical  practices  and  Trauma  Centers  in  the  North-East  US  and  Canada.  Twenty 
surgeons  participated  in  development  and  initial  validation  testing  of  the  performance 
metrics  for  axillary  (AA) ,  brachial  (BA)  femoral  artery  (FA)  exposures  on  unpreserved 
cadavers.  We  next  assessed  performance  of  86  additional  surgeons  in  trauma  core 
procedural  skill  using  the  metrics  we  developed  including  a  100-140  item  checklist,  the 


Individual  Procedure  Score  (IPS),  Global  Rating  Scores  (GRS)  and  errors:  critical 
technical  errors  (CTE) ,  management  errors  (CME)  and  morbidity  errors  (ME). 

The  surgeon  cohorts  studied  were: 

Group  1  =  10  expert  trauma  surgeons  (mean  18  years  in  practice)  and  10  untrained 
surgical  residents  performed  the  four  trauma  core  competency  procedures  to  develop  and 
initially  test  the  performance  metrics.  Analysis  of  Group  1  surgeon  performance  informed 
the  measurement  instruments  (IPS,  GRS,  errors). 

Group  2  =  40  Resident/Fellows  from  13  Surgical  Training  Programs  in  MD  ,  DE,  PA, 
DC,  VA  were  evaluated  before  and  within  4  weeks  of  taking  the  ASSET  Course.  38  out 
of  40  returned  to  be  re-evaluated  between  12-  18  months  (mean  1.2  years)  later.  This  met 
a  priori  sample  size  calculations  indicating  that  >90%  retention  of  Group  2  enrolled 
surgeons  would  be  required  to  address  skill  degradation  with  time  since  training. 


Group  3  =  35  surgeons,  from  a  variety  of  surgical  specialties  who  took  the  ASSET 
Course  mean  2.5  years  earlier  were  evaluated  . 

Group  4=10  Expert  practicing  (mean  14  years  and  all  different  to  Group  1  experts) 
Attending  Trauma  Surgeons  from  6  different  Level  1  US  trauma  centers. 

General  Methods:  We  audio-video  recorded  and  evaluated  Groups  2-4  with  two  trained 
co-located  evaluators  while  they  performed  the  4  American  College  of  Surgeons  core 
competency  trauma  procedures  (AA,BA,FA,  FAS)  on  unpreserved  cadavers  and  realistic 
models.  Evaluations  included  the  components  of  Non-Technical  and  Technical  skills 
described  by  IPS  checklists  developed  for  each  procedure,  the  GRS,  error  assessments 
and  procedure  time,  all  embedded  in  a  standardized  script.  For  data  collection,  we  used  a 
custom  Mobile  application  (App)  running  on  an  Android  Tablet  with  a  built  in  timer.  The 
App  ensured,  by  a  software  prompt,  100%  data  entry  of  the  100-  140  different  evaluation 
points.  We  identified  7  features  of  performance  that  are  amenable  to  training 
interventions.  These  were:  1)  Anatomy  (Landmarks;  Skin  Incision;  Procedural  Steps; 
Correct  Vessel  identification),  2)  Knowledge  of  procedure,  3)  Management/Indications, 
4)  Technical  Skills,  5)  Errors  (Critical  technical;  Critical  Management;  Morbidity),  6) 
Error  recovery,  7)  Time  to  complete  procedure.  We  defined  a  Trauma  Readiness  Index 
(TRI)  as  the  sum  of  all  IPS  scores  for  all  vascular  procedures.  Linear  mixed  models 
(including  cadaver  body  characteristics  and  anatomic  variants;  gender  of  surgeon; 
evaluator  status:  anatomist  v  physician;  surgeons  interval  experience),  multivariate 
(MANOVA)  and  univariate  analyses  were  used  to  compare  mean  differences  between 
Groups2-4. 

FINDINGS: 

Results:  Question  1:  Do  core  trauma  surgical  skills  show  improvement  with  training? 


Improvement  across  all  metrics  was  significant  (p  <0.001)  with  procedural  skills 
training.  Among  the  component  parts  of  the  IPS  in  Group  2  surgeons,  within  4  weeks 
after  ASSET  training:  procedural  steps  improved  57%,  Anatomic  skills  increased  43%, 
Trauma  Readiness  Index  increased  25%  and  time  to  proximal  control  by  passage  of  a 
double  vessel  loop  around  the  artery  decreased  by  an  average  of  2.5  minutes.  ASSET 
training  showed  benefit  regardless  of  surgical  resident  year  of  training,  but  was 
influenced  by  prior  experience.  No  change  occurred  in  trauma  patient  management  (not 
specifically  taught  in  the  ASSET  course).  GRS,  errors  (CTE,  CME,  ME)  and  error 
recovery  were  all  significantly  improved  with  ASSET  training. 

Errors:  Large  variation  in  errors  occurred  both  before  and  after  ASSET  training.  At  an 
IPS  cut-off  of  60%,  CTE  and  CME  increased  exponentially.  Overall,  CTE  were 
significantly  reduced  with  training. 

Performance:  IPS  scores  showed  great  variability  (range  =  <  50%  to  >  80%)  among  40 
Group  2  surgeons  after  ASSET  training,  5  of  whom  failed  to  show  any  performance 
benefit  from  training.. 

FAS  performance:  IPS  was  variable  and  had  the  most  technical  errors  (failure  to 
completely  decompress  all  four  compartments)  by  all  enrolled  surgeons.  50%  of  Group  2 
surgeons  failed  to  adequately  decompress  a  single  FAS  compartment  before  and  50% 
successfully  decompressed  all  4  FAS  compartments  after  ASSET  training. 

Results:  Question  2:  Which  components  among  these  skills  most  benefit  from  training? 

Most  beneficial  training  performance  improvements  were:  correct  incision 
landmarks,  procedural  steps  reduced  time  for  vascular  control,  error  reduction  and 
increased  error  recognition.  We  retained  100%  (40/40)  Group  2  surgeons  for  follow-up 
evaluation  within  4  weeks  of  training.  Group  2  had  significantly  higher  overall  IPS,  fewer 
errors  and  better  error  recovery  for  the  3  vascular  control  procedures,  immediately  after 
taking  the  ASSET  course  than  pre-training.  After  ASSET  training,  as  judged  by  IPS 
components,  twenty  one  of  the  40  Group  2  surgeons  came  within  one  nearest  neighbor 
classifier  (about  one  standard  deviation)  of  Group  4  expert  performance  in  identification 
of  the  correct  incision  landmarks  and  procedural  steps  for  the  vascular  control 
procedures.  Seven  of  the  Group  2  surgeons  after  taking  the  ASSET  course  did  not 
improve  correct  incision  landmarks  and  procedural  steps  for  vascular  control  sufficiently 
to  leave  the  one  nearest  neighbor  classifier  cohort  of  pre-training  performance 

When  performance  was  reviewed  without  bias  (blind  review  of  randomly  ordered  before 
and  after  training  video  clips  by  5  trained  evaluators)  there  was  near-perfect  rater 
agreement  between  video  and  co-located  evaluators  for  anatomy,  technical  skills  and 
readiness  to  perform  the  procedure  independently,  recognition  of  errors  and  overall  rater 
evaluations  for  all  four  procedures. 


Training  interventions:  tracking  IPS  component  scores  and  errors  predicted  need  for 
refresher  training.  On  the  basis  of  plotting  IPS  component  scores  against  Pre-training 
score,  the  training  benefit  is  greater  for  correct  landmarks  than  procedural  steps. 

Performance  on  one  core  procedure  predicting  performance  on  another:  the  fit  on  a 

contour  plot  of  IPS  on  a  single  vascular  control  procedure,  can  predict  (R=0.6)  the  IPS 
performance  metric  on  the  other  two. 

Results:  Question  3:-  Does  training  reduce  the  occurrence  of  error? 

Critical  Technical  Errors  were  reduced  and  error  recovery  increased  with  training. 

Only  13/38  (34%)  of  Group  2  surgeons  performed  the  vascular  control  procedures  free  of 
CTE  (vessel  loop  around  incorrect  structure  or  fail  to  control  <20  minutes).  Five  of  these 
same  surgeons  (12.5%)  had  a  disproportionate  incidence  of  consistent  repeated  error 
despite  training.  For  the  remaining  surgeons  there  was  a  significant  decrease  in  the 
incidence  of  error  from  60%  to  about  20%  in  the  same  surgeons  with  ASSET  training. 
There  was  also  an  increase  in  self-recognition  and  correction  of  technical  errors  related  to 
incorrect  identification  of  anatomic  structure  and  failure  to  complete  vascular  control 
within  20  minutes.  When  CTE  and  CME  were  plotted  against  years  since  ASSET 
training,  only  41%  (35/85)  Group  3  surgeons,  34%  (13/38)  Group  2  and  50%  (5/10) 
Group  4  experts  performed  the  vascular  control  procedures  free  of  either  CTE  or  CME. 
IPS  scores,  but  not  GRS,  were  correlated  with  technical  errors  in  preventing  hemorrhagic 
exsanguination.  CTE  reduction  in  Group  2  surgeons  persisted  in  follow-up  evaluation 
(mean  of  1.2  years  later),  and  was  accompanied  by  increased  error  recognition  and 
recovery. 


Results:  Question  4:  Can  we  identify  a  time  since  training  when  skills  may  need  to  be 
refreshed? 

MANOVA  generally  showed  interval  experience  NOT  time  since  training  was 
correlated  with  lower  IPS  and  more  errors  in  Group  3  surgeons.  The  35  Group  3 
surgeons  (mean  2.5  years  after  training)  showed  a  large  variety  of  interval  experience 
since  taking  the  ASSET  course  (orthopedic,  pediatric,  plastic,  general  surgeons,  critical 
and  acute  care  surgeons  etc).  CTE  and  CME  were  significantly  greater  and  error  recovery 
less  than  either  the  Group  2  surgeons  immediately  and  mean  1 .2  years  after  training. 
Group  3  made  more  “errors”  (Group  2  =  2.4±0.66,  Group  3  =  4.1+1.01,  Group  4  = 

2. 2+0.7  errors/surgeon,  p<0.05),  indicating  that  fewer  procedures  were  correctly 
completed.  Errors  increased  and  error  recovery  decreased  in  Group  3  surgeons  with 
longer  time  since  ASSET  training.  Group  4  expert  surgeons  were  better  performers 
overall  and  had  least  errors  among  Groups.  Group  2  surgeons  had  significantly  higher 
overall  IPS,  fewer  errors  and  greater  error  recovery  after  taking  the  ASSET  course  than 
Group  3  surgeons. 


Years  since  ASSET  Training:  Regression  lines  for  the  TRI  plotted  against  years  since 
ASSET  training  in  all  85  surgeons  show  no  fall  off  in  performance  and  no  threshold  of 
time  since  ASSET  Training  after  which  skill  degradation  was  detected  by  the  IPS  or  GRS 
metrics.  Analysis  with  a  linear  mixed  model  accounting  for  cadaver  habitus,  interval 
experience  and  evaluator  reliability  reveals  significantly  more  critical  errors  and 
decreased  error  recognition  and  recovery  in  the  Group  3  surgeons  peaking  mean  2.5  year 
after  ASSET  Training.  Four  GRS  and  one  overall  evaluator  rating  did  not  reflect  the  skill 
degradation  or  CTE  detected  by  IPS. 


Fasciotomy  Results  Different  from  Vascular  Procedures:  Training  did  increase  the 
number  of  LE  Fasciotomy  compartments  decompressed.  However,  only  20/40  Group  2 
surgeons  succeeded  in  decompressing  all  4  compartments  immediately  after  training. 
Group  3  surgeons  decompressed  significantly  fewer  FAS  compartments  than  Group  4  or 
Group  2  surgeons.  The  majority  of  85  surgeons  evaluated  after  training,  including  4/10 
experts,  failed  to  decompress  at  least  one  FAS  compartment.  FAS  was  the  most  error- 
prone  procedure  because  of  incomplete  decompression  of  the  anterior  or  deep  posterior 
compartments.  FAS  is  a  sentinel  trauma  procedure,  as  it  both  demonstrates  the  benefits  of 
ASSET  training  and  it  detects  the  occurrence  of  skill  degradation. 


Results:  Question  5:  Can  hyper-realistic  physical  models  of  each  procedure  replace  the  unpreserved 
cadavers  used  in  this  study? 

The  physical  model  tested  cannot  replace  cadaver  for  competency  evaluations  because 
on  the  model  IPS  scores  are  higher,  there  are  1/3  less  errors  and  time  to  complete 
procedures  is  half  that  of  the  same  surgeons  performing  the  same  procedures  on  the 
cadaver.  For  every  procedure  and  for  every  group  of  surgeons  a  shorter  time  is  required 
to  find  the  artery  or  decompress  the  compartments  because  models  fail  to  capture  the 
complexity  and  variability  of  the  human  cadaver.  The  model  facilitates  discrimination,  so 
that  anatomical  structures  are  much  easier  to  recognize.  The  model-based  assessment 
could  not  identify  skill  degradation,  as  the  same  surgeons  have  fewer  errors  in  the  models 
than  in  the  cadaver. 


RECOMMENDATIONS: 

1)  IPS  scoring  should  be  used  during  ASSET  course  training  for  AA,  BA,  FA,  and  FAS  to 
provide  performance  feedback  and  formative  evaluations  to  determine  readiness  for 
surgeon  deployment. 

2)  Remote  evaluation  of  video  recorded  performance  of  surgeons  operating  on  physical 
models  should  be  tested  as  a  surrogate  alternative  to  cadaver  use  as  a  mobile  training 
platform  not  exclusively  for  formative  assessment. 


3)  Targeted  skills  refresher  should  include  correct  incision  landmarks,  procedural  steps 
and  structural  anatomy  for  AA,  BA ,  FA  and  FAS  procedures. 

4)  The  Mobile  App  should  be  fielded  to  centers  holding  ASSET  courses 

5)  The  database  and  video  recordings  associated  with  this  study  should  be  de-identified 
and  made  available  for  other  users  and  training  uses 


DELIVERABLES: 

1)  Software  to  evaluate  IPS  for  3  vascular  procedures  and  lower  extremity  fasciotomy. 

2)  A  Mobile  Android  App  for  data  collection 

3)  Physical  Models  evaluated  for  AA,  BA,  FA  and  FAS  procedures 

4)  A  Mobile  Platform  for  evaluation  and  analysis  of  vascular  control  and  fasciotomy 
procedures 
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Phase  I  -  Preliminary  investigations,  TRR  audit  modification,  and  validation  of  Advanced 
Surgical  Skills  for  Exposure  in  Trauma  (ASSET)  Performance  testing  methods 

Task  la  )  IRB  submission;  Kick— off  meeting  of  clinical  and  research  staff,  months  0-2.  Due  Days 
from  Award  (DFA):  60  days;  Acceptance  Criteria  (AC):  Meeting  minutes  and 
presentation  materials,  IRB  approval;  Percentage  of  Cost  ( POC ):  1% 

•  Kick-off  Meeting  February  14th  2013  Agenda  and  Minutes  Recorded  (Appendix  44).  IRB 
Protocol  submitted  to  UM  IRB  (13  Dec  2012)  and  approved  (8  Jan  2013). 

Task  lb)  Acquisition  of  hardware,  Trauma  Reception  and  Resuscitation  (TRR)  software  and 
equipment;  months  0-2.  DFA:  60  days;  AC:  Equipment  etc.  acquired;  POC:  5% 

•  See  attached  Invoices  for  Acquisition  of  surgical  hardware  for  surgeons  to  perform 
procedures  during  evaluations  (Appendix  45).  See  Attached  TRR  Acquisition  for 
software  and  equipment  (Appendix  46). 

Task  lc)  Analyze  data  from  self-assessments  provided  by  >600  past  ASSET  trainees,  months  0-3; 
DFA:  90  days;  AC:  Statistical  analysis  of  dataset;  POC:  3%. 

•  We  compared  self-reported  confidence  of  participants  (n=523)  with  surgical  tasks 
(n=47)  at  baseline  and  directly  after  ASSET  training  to  examine  the  effect  of 
training.  All  surgeons  recorded  improved  confidence  in  all  five  anatomic  body 
regions  after  ASSET  training  (p<0.0001).  Following  the  course,  surgeons 
reported  a  high  confidence  level  in  78%  of  the  47  procedures.  Residents/fellows 
achieved  the  greatest  improvement  in  confidence  levels.  This  study  highlights  the 
broad  positive  impact  of  the  ASSET  course  on  trauma  surgical  skills.  An 
objective  performance  measure  of  surgical  skills  would  be  valuable  for  future 
course  development  (Appendix  47).  See  important  supporting  data  (Figures  1-3;  Tables 
1-3). 


Task  Id)  Audio-visual  (A  V)  recording  of  “thinking  out  loud,  ”  and  responses  to  questions  on 
technical 

and  non-technical  skills  and  fidelity  of  physical  models  vs  cadaver  during  ASSET  procedures 
by  10  expert  surgeons  and  10  surgeons  without  prior  ASSET  training,  months  3-7.  DFA:  210 
days;  AC:  Completion  AV recording  and  AV data  collection  synthesis;  POC:  10% 

•  See  attached  Invoice  for  Acquisition  of  physical  models  for  use  during  Expert  and 
Novice  performance  evaluations  (Appendix  48). 

•  We  developed  an  index  (Trauma  Readiness  Index)  to  quantify  surgical  performance  and 
competence  derived  from  knowledge,  procedural,  and  technical  skills  components. 

•  To  establish  a  “gold  standard”  range  for  expert  performance  suitable  for  comparisons 
with  surgeons  in  later  phases,  it  was  necessary  to  assess  10  additional  expert  surgeons 
using  the  same  skills  assessment  tool.  Data  from  the  10  expert  surgeons  has  been 


compared  with  Phase  2  and  Phase  3  performance  assessment  data.  These  data  provide  a 
benchmark  to  compare  the  magnitude  of  skills  performance  improvement  after  training 
and  degradation  in  the  years  following  training  (Appendix  2,  Figure  1).  Expert  and 
Novice  performance  evaluations  were  completed  in  October  2013. 

•  Results  from  assessing  the  performance  of  Novice  surgeons  with  the  Trauma  Readiness 
Index  metric  were  presented  at  MHSRS  (August  2014).  We  found  that  this  metric 
discriminated  expert  from  novice  performance  in  both  technical  and  non-technical  skills 
with  excellent  interrater  reliability  (Appendix  2). 

•  The  importance  of  assessing  Novice  performance  utilizing  the  technical  and  non¬ 
technical  skills  identified  by  the  expert  surgeons  was  also  presented  as  an  abstract  at 
Association  of  Military  Surgeons  of  US  (AMSUS)  (Society  for  Federated  Health 
Professionals)  in  Dec  2014.  We  found  that  expert  surgeon  technical  skill  metrics  provide 
a  reliable  technical  skill  assessment  for  less  experienced  surgeons.  It  distinguished  Expert 
from  Novice  surgeons  with  excellent  inter-rater  reliability  (Appendix  1). 

Task  le)  Revise  all  conventional  assessment  instruments  in  collaboration  with  the  participants. 

•  This  aspect  of  Task  le)  was  accomplished  by  April  2013.  During  the  “thinking  out  loud” 
by  the  10  experts,  several  key  points  became  apparent  that  were  then  noted  and  included 
in  possible  discriminators.  A  consensus  meeting  of  the  experts  occurred.  Draft  evaluation 
criteria  were  developed  and  then  tested  on  10  novice  (2nd  to  fourth  year  surgical 
residents).  With  minor  iterations  occurring  in  the  content  and  format  of  the  evaluations  as 
each  successive  novice  candidate  was  evaluated. 

Task  le  cont. )  Establish  key  steps  and  landmark  evaluation  points  for  the  ASSET  procedures  from 
A  V  records,  months  6-7.  DFA:  210  days;  AC:  Revised  assessments,  ASSET  steps  and 
landmarks  defined;  ROC:  6% 

•  An  Evaluator  Training  Handbook  and  training  videos  were  developed  before  inter-rater 
reliability  testing  (Appendix  42;  Handbook  and  Videos). 

•  A  Script  was  finalized  for  each  of  the  four  procedures.  The  Knowledge  Content  and 
Technical  Skills  assessments  were  finalized  so  that  one  script  covered  all  four  procedures 
with  breaks  between  procedures.  The  breaks  allow  the  sequence  of  the  procedures  to  be 
changed  so  that  ‘carry-over’  between  before  and  after  ASSET  training  was  minimized.  In 
addition  this  break  was  necessary  so  that  one  candidate  would  not  hear  the  answers  given 
or  see  the  procedure  being  performed  by  another  nearby  candidate  as  might  occur  if  they 
were  doing  the  same  procedures,  at  the  same  time  alongside  each  other.  We  also  have 
included  a  consent  form  for  the  participants  to  sign  (Appendices  41,  43,  and  49;  Script, 
Script  Slides,  Video  Evaluation  Sheet,  and  Consent  Form). 

Task  If)  Modify  TRR  software  to  include  these  points,  and  conduct  inter-rater  reliability  by 

multiple  expert  reviewers  of  ideal  and  non-ideal  ASSET  procedure  performance,  mon  ths 
5-9.  DFA:  270  days;  AC:  TRR  Software  modified  and  TRR  Performance  Audit  tool 
validated;  ROC:  18% 

•  Major  modifications  of  TRR  software  were  accomplished  by  November  2013,  minor  re¬ 
modifications  were  completed  by  April  2014.  The  technology  was  implemented  for 


evaluations  21  April  2014.  A  training  module  was  developed  for  evaluators  (Appendix 

4). 

•  Inter -Rater  reliability  testing  using  5  expert  reviewers  of  80  video  records  and  the 
evaluations  described  above  (under  Task  le)  is  summarized  below  in  an  Abstract 
Submitted  to  the  American  College  of  Surgeons  for  consideration  to  be  presented  at  their 
Annual  Scientific  Meeting.  For  this  Abstract  each  of  5  experts  reviewed  video  recordings 
of  all  the  four  procedures  (Axillary,  Brachial,  Femoral  Artery  Exposure  and  Lower 
Extremity  Fasciotomy)  for  all  10  experts  and  all  10  novices).  Interrater  reliability  was 
assessed  using  intra-class  correlation  coefficient  (ICC).  The  ICC  can  discriminate 
expert  from  novice  performance  for  the  four  surgical  procedures  evaluated 
through  the  use  of  discriminating  performance  characteristics.  These 
characteristics  may  be  useful  for  objective  surgical  skill  assessment  (Appendix  50). 
An  example  of  an  Expert  performance  of  an  Axillary  Artery  exposure  can  be  found  in  the 
attached  digital  file  (Appendix  51). 

•  The  mobile  platform  was  described  and  presented  as  a  poster  at  MHSRS  in  August  2014. 
This  poster  describes  the  head  camera,  pan/tilt/zoom  camera,  audio  capture  and  Android 
software  used  for  non-intrusive  skills  assessment  with  the  ability  for  remote  evaluation 
(Appendix  5). 


Phase  II:  Using  the  revised  and  validated  ASSET  Testing  tools  developed  in  Phase  I  (as 
described  in  Task  le),  examine  the  efficacy  of  the  ASSET  training  curriculum  on 
acquisition  and  retention  of  ASSET  skills,  including  the  relative  efficacy  of 
unpreserved  cadaver  versus  selected  non-live-tissue  models  in  skills  training. 

•  The  relative  efficacy  of  unpreserved  cadaver  versus  selected  non-live -tissue  models  in 
skills  training  was  assessed  using  the  attached  questionnaire  given  to  participating 
surgeons  (Appendix  6).  In  addition,  participants  were  issued  another  questionnaire  used 
to  compare  the  unpreserved  cadaver  to  a  live  patient  (Appendix  7).  We  found  that 
participants  considered  cadavers  to  be  more  realistic  and/or  useful,  but  the  models  still 
received  favorable  ratings.  These  data  we  presented  at  the  MHSRS  2015  (Appendix  29). 

•  Each  surgeon  was  asked  to  rate  their  confidence  for  performing  vascular  surgery  in  the 
upper  and  lower  extremities  before  and  after  each  evaluation  using  the  attached 
questionnaire  (Appendix  8). 

Findings  indicated  that  the  confidence  of  senior  residents  in  their  ability  to  perform  the  3 
vascular  procedures  and  lower  extremity  fasciotomy  was  significantly  higher  than  their 
skills  performance  evaluations.  These  data  were  compiled  and  presented  at  ASC 
(Academic  Surgical  Congress)  in  February  2015  and  a  manuscript  submitted  in  January 
2015  (Appendix  9). 

Task  2a)  Train  forty  (in  cohorts  of  10)  ASSET-untrained  surgeons:  test  base-line  skills,  provide 
ASSET  course,  do  post-test,  months  10-17.  DFA  510  days;  AC:  training  and  Phase  1 
assessments  complete;  POC:  15% 


We  completed  enrollment,  baseline  before -training  skills  assessment,  ASSET  course 
training,  and  post -training  assessments  for  39  of  the  proposed  40  ASSET  naive  surgical 
residents  Sept  3  2014  (Figure  2).  The  final  40th  resident  will  complete  the  post-training 


assessment  by  March  16  2015.  A  last  minute  enrollment  drop-out  necessitated  this 
substitution  and  delay  in  completion. 

•  To  date,  five  additional  abstracts  and  two  manuscripts  have  been  completed  detailing  the 
assessment  of  surgical  skills  for  before  and  after  Paining  co-located  evaluations  and 
remote  video  review.  This  includes  a  manuscript  was  accepted  for  publication  in  the 
Journal  of  Trauma  detailing  the  performance  evaluations  before  and  after  training 
(Appendices  9  -  15). 

Task  2b)  Mid-term  review  meeting  with  investigators  and  consultants  -  2  days  in  month  18.  DFA: 
540  days;  AC:  meeting  minutes  and  presentation  materials  as  appropriate;  POC:  1% 

•  Midterm  In  Progress  Review  (IPR)  was  conducted  August  2014  in  Orlando  Florida.  The 
presentation  slides  and  subject  matter  expert  reviews  are  attached  (Appendices  16  a  &  b). 

Task  2c)  Forty  surgeons  from  2a)  perform  4  ASSET  procedures  in  random  sequence  on  physical 
model  and  cadaver ,  months  11-18.  DFA:  540  days;  AC:  assessments  for  physical  model 
v  cadaver;  POC:  10% 

•  See  response  to  Task  2a)  above 

Task  2d)  Revaluate  2b/2c  surgeons  at  either  12  (n=20)  or  18  months  (n=20)  on  physical  model  & 
cadaver.  DFA:  990days;  AC:  TRR  Performance  Audit  records  and  other  performance 
assessments;  POC:  15% 

•  We  were  able  to  schedule  and  complete  38  of  the  40  follow-up  evaluations. 


Phase  III:  Examine  various  aspects  of  skills  degradation  over  time,  including  comparison  of 
skills  degradation  among  40  surgeons  participating  in  past  ASSET  courses  (cadaver 
model  training  only)  and  those  participating  in  the  study-based  ASSET  training 
curriculum. 

Task  3a)  Recall  and  retest  previously  ASSET-trained  surgeons  on  cadaver  at  intervals  of  2-5 
years  from  original  training,  months  11-30.  DFA:  mon  ths  32-36  DFA;  AC:  Repeat 
ASSET  procedures  in  40  previously  trained  surgeons.  Complete  skills  assessments  as 
originally  administered  and  TRR  Performance  Audit;  POC:  15% 

•  35  surgeons  Pained  2  to  5  years  ago  were  evaluated.  Follow-up  evaluations  concluded  2 
October  2015. 

•  Three  abstracts  were  submitted  to  MFISRS  for  2015  using  preliminary  retention  data 
comparing  skills  levels  of  the  ASSET  alumnae  to  the  39  pre  and  post-training  surgical 
resident  and  8  expert  scores  (Figure  1  &  3;  Table  2). 

•  During  37  of  the  40  Phase  2  follow-up,  21  of  the  Phase  3  evaluations,  and  2  of  the  Expert 
evaluations,  we  implemented  an  additional  fifth  procedure  to  their  assessment:  the  carotid 
artery  exposure.  This  was  employed  in  an  effort  to  eliminate  practice  bias  since 
participants  have  not  been  previously  exposed  to  this  procedure  during  earlier  evaluations 
(2  October  2015)  (Appendix  52;  Script  with  Carotid  Artery  Procedure  addition  and 
updated  Powerpoint  slides). 

Task  3b)  Data  analysis;  draft  paper  and  present  results,  37  months  DFA;  AC:  Fined  report 


acceptance;  POC  1  % 


•  Data  analysis  and  paper  drafting  involved  error  analysis,  skill  degradation,  transcript 
analysis,  cadaver  vs.  model,  blind  video  analysis,  literary  review  of  trauma  surgical 
technical  skills,  item  analysis,  and  longitudinal  analysis. 

•  Created  and  submitted  Final  Report  (15  Mar  2016). 


Key  Research  Accomplishments 

•  Utilized  the  comprehensive  database  of  video  clips  demonstrating  surgical  technique 
from  Phase  1  Expert  and  Novice  subjects  and  the  training  manual  to  train  a  total  of  23 
additional  surgeons  and  anatomists  as  reviewers  to  evaluate  surgical  skill  and  technique 
(15  August  2013  -  4  June  2015). 

•  Submitted  2013  Quarterly  Reports  for  the  periods  of  15  February  2013  -  15  May  2013, 

15  May  2013  -  15  August  2013,  15  August  2013-15  November  2013,  and  15  November 
2013- 15  February  2014. 

•  Kristy  Pugh  established  her  role  as  the  project’s  research  assistant  (1  April  2014  -  14  Feb 
2016)  (Table  4). 

•  Continued  collaboration  with  TRR  software  developers  from  Swinburne  Australia  to 
refine  collected  evaluation  data  (7  April  2014  -  14  Feb  2016). 

•  Successfully  employed  the  evaluation  application  and  conducted  co-located  surgical 
skills  evaluations  solely  on  the  tablets  using  the  RASP  Application  in  real-time  (2 1  April 
2014  -  2  October  2015)  (Appendix  4). 

•  Attended  the  annual  meeting  of  the  American  Association  of  Anatomists  at  FASEB  and 
presented  results  from  the  ASSET  historical  dataset  analysis  (27  April  2014)  “The  Assets 
of  ASSET :  Improving  surgical  performance  through  an  anatomy  and  skills  review  course 
for  surgeons.”  We  found  that  after  taking  the  ASSET  course,  there  were  significant  gains 
in  confidence  scores  for  surgeons  of  all  specialties.  This  demonstrates  the  value  of 
continuing  education  in  applied  anatomy  for  clinical  practice  (Appendix  17). 

•  Budget  modification  submitted  and  accepted  to  properly  reallocate  funds  (June  2014 
(submitted)  -  2014  (accepted))  (Appendix  3). 

•  Completed  the  baseline  evaluations  of  40  of  the  40  Phase  11  surgeons  (before  receiving 
ASSET  Paining)  (11  February  2015)  and  39  of  the  40  surgeons  post-training  (3 
September  2014). 

•  Completed  review  of  anonymized  videos  for  the  first  12  completed  pre  and  post 
evaluations  (1 1  September  2013). 

•  Cris  Imle  has  established  herself  as  a  full-time  schedule  coordinator  to  find,  contact, 
recruit,  and  schedule  evaluations  for  Phase  II  surgeons  12  or  18  month  follow-up  and 
Phase  III  surgeons  until  the  completion  of  the  participant  evaluations  (1  October  2014-9 
October  2015). 

•  UM  IRB  annual  continuing  review  submitted  (March  2013,  14  November  2014,  and 
August  2015)  and  accepted  (17  November  2014  and  4  August  2015). 

•  Held  a  consultants  meeting  with  experts  Valerie  Shalin  and  James  Shanteau  to  address 
how  we  assess  and  understand  expertise  and  examine  the  possibility  of  new  metric 
methods  for  analysis  (20-21  Nov  2014). 


Presented  an  abstract  at  the  AMSUS  2014  (3  December  2014)  reporting  the  assessment 
of  technical  skills  through  video  evaluations  “Evaluation  of  individual  surgeon  technical 
skills  during  four  emergency  procedures”  (Appendix  1). 

Submitted  2014  Annual  Report  for  the  period  of  15  February  2014  to  15  February  2015. 
Submitted  Quarterly  Reports  for  the  periods  of  15  February  2014  to  15  May  2014,  15 
May  2014  to  15  August  2014,  15  August  2014  to  15  November  2014,  and  15  November 
2014  to  15  February  2015. 

Recruited  a  substitute  surgeon  for  the  Phase  II  drop  out  -  thus  completing  the  enrollment 
of  40  of  the  40  Phase  II  participants  (12  January  2015). 

Presented  abstract  at  Eastern  Association  of  Surgery  for  Trauma  (EAST)  (13  January 
2015)  regarding  the  preliminary  results  of  Trauma  Readiness  Index  for  pre  and  post¬ 
training  data  “Development  of  a  Trauma  Readiness  Metric  Score  for  Surgeons” 
(Appendix  2). 

Manuscript  associated  with  abstract  submitted  for  the  EAST  2015  meeting  was  accepted 
for  publication  in  Journal  of  Trauma  (16  January  2015)  “Development  and  Validation  of 
Trauma  Surgical  Skills  Metrics:  Preliminary  Assessment  of  Performance  after  Training” 
(Appendix  2). 

Abstract  accepted  for  presentation  at  the  Federated  American  Societies  of  Experimental 
Biology  (16  January  2015)  “Surface  Anatomy  in  the  Performance  of  a  Eower  Extremity 
Fasciotomy  before  and  after  Training.”  We  found  that  after  lower  extremity  fasciotomy 
training,  surgical  residents  improved  in  their  landmark  identification,  incision  placement, 
and  successful  4  compartment  decompression  (Appendix  15). 

Completed  base-line  evaluation  of  40  of  the  40  proposed  Phase  II  surgical  residents  (10 
February  2015). 

One  surgeon  was  unable  to  complete  their  post-training  evaluation  due  to  scheduling 
conflicts.  We  were  able  to  recruit  a  replacement  Phase  II  surgeon  and  completed  both 
their  baseline  and  post-training  evaluations  (11  February  2015  -  16  March  2015). 
Completed  evaluations  of  10  of  10  Expert  surgeons  with  the  same  metrics  used  to  assess 
the  Phase  II  and  III  surgeons  to  create  a  systematic  “expert  range”  of  surgical  skill  for 
comparison  with  other  study  participants  (8  June  2015)  (see  Figures  1  &  3). 

Completed  evaluations  on  40  of  the  40  Phase  II  participants  before  and  after  ASSET 
training  (16  March  2015). 

Second  budget  modification  submitted  and  accepted  to  pay  for  additional  cadaver  use. 
(March  2015  (submitted)  -  November  2015  (accepted))  (Appendix  21). 

Held  a  meeting  with  Advisory  Board  Member,  Prof  Nick  Sevdalis  (Fondon  UK)  in 
Seattle  at  the  ASE  meeting  (April  21-25  2015)  to  address  how  to  validate  our  Individual 
Procedure  Score  (IPS)  performance  metric  and  to  update  Prof  Sevdalis  on  project 
progress. 

Presented  a  poster  and  2  podium  presentation  at  the  Association  for  Surgical  Education 
(ASE)  (April  21st  and  22nd  2015)  “Management  of  Vascular  Trauma  by  Senior  Surgical 
Residents:  Perception  Does  Not  Equal  Reality”,  “Mobile  Platform  for  Assessing 
Emergency  Trauma  Surgical  Skill  Performance”,  and  “Assessment  of  surgical  anatomy 
skills  in  upper  and  lower  limb  vascular  control  and  before  and  after  training.”  We  found 
that  our  tablet  is  able  to  capture  surgeon  evaluation  metrics  during  assessments  with  little 
intrusion  and  without  paper  copies,  and  that  ASSET  training  may  help  accelerate 
acquisition  of  emergency  surgery  specific  skills  to  compensate  for  shortened  training 
hours  (Appendices  9,  11,  and  13). 


Presented  poster  at  Federated  American  Societies  of  Experimental  Biology  (FASEB)  for 
the  American  Association  of  Anatomists  “Surface  anatomy  in  the  performance  of  a  lower 
extremity  fasciotomy  before  and  after  training”  (April  30,  2015)  (Appendix  15). 
Completed  17  of  the  40  Phase  111  ASSET  alumni  evaluations.  Scheduled  a  further  12 
Phase  111  surgeons  for  skill  retention  evaluation  (12th  May  2015). 

Made  significant  progress  in  re -programming  RASP  Application  for  Android  Tablet  to 
allow  addition  of  other  surgical  procedures  and  for  changes  to  be  made  to  screens  in- 
house  rather  than  through  sub-contractor  (Appendix  4). 

Presented  poster  at  American  Association  of  Clinical  Anatomists  2015  meeting 
“Anatomic  knowledge  increases  after  participation  in  ASSET  training”  (12  June  2015). 
We  found  that  anatomic  knowledge  benefits  significantly  from  ASSET  training 
(Appendix  25). 

Dr.  Guinevere  Granite,  PhD,  has  established  her  position  as  Research  Coordinator  (1  July 
2015  -  14  Feb  2016)  (Table  4). 

Mi's.  Nyaradzo  Longinaker  has  established  her  position  as  graduate  research  assistant  and 
statistician  to  manage  the  collected  data  for  the  project  and  organize  it  for  use  in  future 
publications  (1  July  2015  -  4  December  2015). 

Manuscript  published  in  The  Journal  of  Trauma  and  Acute  Care  Surgery  (July  2015) 
“Development  and  validation  of  trauma  surgical  skills  metrics:  Preliminary  assessment  of 
performance  after  training.”  Using  our  Trauma  Readiness  Index  as  a  single  performance 
score  that  combines  completion  time  and  performance  assessment  metrics,  we  can  detect 
improvement  in  specific  procedure  steps,  overall  procedure  completion  time,  and 
anatomic  knowledge  after  the  ASSET  course  (Appendix  10). 

Manuscript  published  online  in  the  Journal  of  Surgical  Education  (23  July  2015)  “Using 
an  Individual  Procedure  Score  Before  and  After  the  Advanced  Surgical  Skills  Exposure 
for  Trauma  Course  Training  to  Benchmark  a  Hemorrhage-Control  Performance  Metric.” 
We  found  that  improved  surface  landmark  knowledge,  obtained  during  the  ASSET 
course,  correlates  with  increased  IPS,  faster  procedures,  more  accurate  incision 
placement,  and  successful  vascular  control  (Appendix  26). 

Presented  4  podium  presentations  at  the  Military  Health  System  Research  Symposium 
(17  -  20  August  2015)  “Performance  of  Combat  Surgical  Skills  before  and  after  ASSET 
training”,  “Are  Physical  Models  Comparable  to  Cadaver  for  Assessing  Combat  Surgical 
Technique?”,  and  “Accurate  Assessment  of  Surgical  Skill  Improvements  after  Training: 
Development  and  Validation  of  Trauma  Surgical  Skills  Metrics,  Preliminary 
Assessment”,  and  “How  successful  is  ASSET  at  training  residents  in  lower  extremity 
fasciotomy  compared  to  experienced  trauma  surgeons?”  Findings  from  these 
presentations  include:  previously  trained  surgeons  in  active  surgical  practice  would 
benefit  from  a  trauma-specific  skills  refresher  course,  combat  surgical  procedure 
performance  degrades  significantly  2  to  5  years  later,  and  correct  identification  of  surface 
landmarks  and  incisions  are  associated  with  improved  vascular  control  performance 
(Appendices  12,  28,  30,  and  33). 

Presented  Plenary  Session  Podium  presentation  as  a  Finalist  for  the  Young  Investigator 
Award  at  the  Military  Health  System  Research  Symposium  (17-20  August  2015)  “How 
successful  is  ASSET  at  training  residents  in  lower  extremity  fasciotomy  compared  to 
experienced  trauma  surgeons?”  (Appendix  28). 

Completed  evaluations  on  38  of  the  40  Phase  II  participants  during  their  follow-up  period 
(12  to  18  months)  following  ASSET  training  (14  September  2015). 


Request  for  Statement  of  Work  modification  submitted  and  accepted  to  add  carotid  artery 
surgical  procedure  to  modified  RASP  Android  application,  and  pay  additional  evaluators 
(submitted  (September  2015)  and  accepted  (January  2016)  (Appendix  35). 

Third  budget  modification  submitted  and  accepted  to  pay  for  additional  cadaver  use. 
(September  2015  (submitted)  -  January  2015  (accepted))  (Appendix  36). 

Completed  random  pre-  and  post-video  review  for  inter-rater  consistency  review. 

Analysis  of  four  evaluators  blind  review  4  pre-  and  post-  procedures  for  each  procedure 
from  upper,  mid,  and  lower  tertiles  of  performance  (September  2015). 

Completed  35  of  the  40  Phase  111  ASSET  alumni  evaluations  (2  October  2015)  (see  Figs. 

1  &3). 

Downloaded  final  data  set  from  Cloud  and  tablet  co-located  data  (3  October  2015). 

After  completing  our  final  participant  evaluation,  we  downloaded  the  complete  raw 
evaluation  data  set  from  the  cloud  and  began  preparing  additional  manuscripts  to  be 
submitted  to  peer-reviewed  surgical  journals  (3  October  2015). 

Hosted  an  advisory  board  meeting  with  Dr.  James  Shanteau,  psychologist  and  expert  on 
expert  performance  analysis,  Dr.  Valerie  Shalin,  psychologist  and  expert  in  cognitive  task 
analysis  and  human  factors,  and  Prof.  Nick  Sevdalis,  an  experimental  psychologist  and 
expert  in  patient  safety  in  hospital  environments  from  King’s  College,  London  (7-9 
October  2015  at  University  of  Maryland,  Baltimore)  (Appendix  22). 

Organized  and  error  checked  the  final  data  set,  created  a  data  dictionary  and  calculated 
IPS  and  TRI  metrics  for  data  set  analysis  for  current  and  future  publications  to  begin 
analysis  for  use  in  current  and  future  publications  (23  November  2015). 

Applied  the  data  dictionary  to  statistical  analyses  for  use  in  current  and  future 
publications  and  began  preparing  additional  manuscripts  to  be  submitted  to  peer- 
reviewed  surgical  journals. 

Presented  at  the  International  Meeting  for  Simulation  in  Healthcare  (1MSH)  (18  January 
2016)  “Acquisition  &  Retention  of  Trauma  Surgical  Skills”.  We  presented  the  following 
findings:  ASSET  training  benefitted  procedural  steps  (57%  increase),  Anatomic  skills 
(43%),  Technical  skills  (25%),  and  readiness  to  perform  vascular  procedures  (28%)  with 
a  mean  decrease  of  2.5  minutes  for  procedure  time.  (Appendix  24). 

Submission  and  acceptance  of  IRB  renewal  (Appendix  35). 

Submitted  Annual  Report  for  the  period  of  15  February  2014  to  15  February  2015. 
Submitted  Quarterly  Reports  for  the  periods  of  15  February  2015  to  14  May  2015,  15 
May  2015  to  14  August  2015,  15  August  2015  to  14  November  2015,  and  15  November 
2015  to  14  February  2016. 

Submitted  abstract  “Critical  Errors  in  Rarely  Performed  Procedures  0.5-5  Years  After 
Training  Among  85  Surgeons”  to  the  International  Anesthesia  Research  Society  (IARS) 
and  the  Association  of  University  Anesthesiologists  (AUA)  2016  meeting  (22  January 
2016)  (Appendix  53). 

Submitted  manuscript  “Sensor-Free  Computer- Vision  Hand-Motion  Entropy  and  Video- 
Analysis  of  Technical  Performance  during  Open  Surgery:  Proof  of  Concept  Report  of 
Methodology  and  Analysis”  for  the  Human  Factors  &  Ergonomics  Society  (HFES)  2016 
meeting  (3  February  2016)  (Appendix  31). 


Reportable  Outcomes 

•  Compiled  an  extensive  surgical  video  library  over  a  range  of  skill  levels  from  head 
camera  and  ceiling  mounted  cameras. 

•  Completed  the  development  and  implementation  of  a  mobile  skills  evaluation  platform 
including  a  metric  tool  in  an  Android-based  software  application. 

•  Twelve  abstracts/podium  presentations/posters  were  presented  at  various  relevant 
professional  meetings  (AMSUS  2014.  ASC  2014.  ASA  2014,  EAST  2014,  FASEB  2014, 
MHSRS  2014,  AACA  2015,  ASE  2015,  FASEB  2015,  MHSRS  2015)  (Appendices  1,  9- 
13,  15,25,27,  28,  and  33) 

•  Submitted  a  pre -proposal  to  continue  to  assess  the  acquisition  and  retention  of  surgical 
skill  by  examining  the  efficacy  of  multiple  skills  and  knowledge  refreshing  methods: 
BA150077  -  “Refreshing  Combat  Surgical  Skills”  (December  2014)  (Appendix  40). 

•  Invited  to  submit  a  full  proposal  of  BA150077  -  “Refreshing  Combat  Surgical  Skills”  (15 
January  2015,  Appendix  19)  and  submitted  a  full  proposal  (30  March  2015,  Appendix 
38). 

•  We  have  completed,  error  checked,  updated  and  used  our  entire  dataset  (including  Phase 
2,  Phase  3,  and  Experts  data)  for  multiple  analyses. 

•  Dr.  Colin  Mackenzie  (PI)  successfully  defended  his  candidacy  for  his  UK  Doctoral 
degree  based  on  this  research  effort.  He  presented  his  thesis  proposal  "Assessment  of 
Surgical  Performance:  Early  Stage  Assessment  "  (November  2015)  (Appendix  20). 

•  Completed  the  (40  of  40)  Phase  II  surgeon  evaluations  by  co-located  evaluators  before 
and  after  ASSET  training  (16  March  2015). 

•  Completed  38  of  the  40  Phase  II  surgeon  follow-up  evaluations  12  to  18  months  after 
ASSET  (21  September  2015). 

•  Completed  35  of  the  40  Phase  III  ASSET  alumni  evaluations  (2  October  2015). 

•  Manuscript  published  in  The  Journal  of  Trauma  and  Acute  Care  Surgery  “Development 
and  validation  of  trauma  surgical  skills  metrics:  Preliminary  assessment  of  performance 
after  training”  (July  2015)  (Appendix  10). 

•  Manuscript  published  online  in  the  Journal  of  Surgical  Education  “Using  an  Individual 
Procedure  Score  Before  and  After  the  Advanced  Surgical  Skills  Exposure  for  Trauma 
Course  Training  to  Benchmark  a  Hemorrhage-Control  Performance  Metric”  (submitted  7 
April  2015,  published  23  July  2015)  (Appendix  26). 

•  Completed  preliminary  analysis  of  tracking  hand  movement  during  open  surgery  using  an 
infrared  imaging  camera  and  analysis  system  (ICI  9640  P,  Infrared  Cameras  Inc., 
Beaumont,  TX)  using  different  colored  gloves  to  indicate  dominant  and  non-dominant 
hand  for  experts  vs.  novice  surgeons.  (July-August  2015). 

•  Received  rejection  letter  for  full  proposal  submission  of  BA150077  -  Titled  Refreshing 
Combat  Surgical  Skills  (20  September  2015). 

•  Completed  collecting  and  recording  new  data  (63  total  evaluations)  involving 
implementation  of  carotid  artery  exposure  as  a  fifth  procedure  during  follow-up 
evaluations  for  Phase  II  participants,  retention  period  evaluations  for  Phase  III 
participants,  and  Expert  participants.  This  was  employed  in  an  effort  to  eliminate  practice 
bias  since  participants  have  not  been  previously  exposed  to  this  procedure  during  earlier 
evaluations  (2  October  2015). 

•  Submitted  Final  Report  for  the  period  of  15  February  2013  to  15  February  2016. 

•  Completed  Principal  Component  Analysis  of  Pre  and  Post  course  co-located  evaluations 
with  complete  data  set  of  38  participants  in  Phase  II  and  35  participants  in  Phase  III. 


•  Performed  analyses  on  completed  data  set  for  use  in  current  and  future  publications. 

•  Continued  to  prepare  and  present  abstracts  and  manuscripts  for  various  aspects  of  data 
analysis  (Appendices  24  and  31). 

•  Modified  the  Android  data  collecting  tool  application  to  allow  for  accurate  storage  of 
evaluation  data  to  the  tablet  before  uploading  it  to  the  Cloud.  This  will  allow  for  revisions 
to  the  data  during  procedure  de -briefing  prior  to  uploading  the  data  to  the  Cloud.  We  also 
adapted  the  Android  data  collecting  tool  application  to  facilitate  creation  of  new 
procedures  to  be  used  during  future  evaluations  (January  -  February  2016). 

•  Modified  our  original  Individual  Procedure  Score  (IPS)  metric  to  exclude  errors  and 
procedure  time,  which  we  designated  as  Adjusted  IPS  for  publications  involving  error 
analysis  and  other  data  analysis  topics. 

•  To  facilitate  comprehension  and  use  of  the  final  RASP  data  set,  we  developed  a  complete 
data  dictionary  for  all  current  and  future  research  team  members. 

•  Applied  our  modified  Individual  Procedure  Score  (IPS)  metric  that  excludes  errors  and 
procedure  time  (designated  as  Adjusted  IPS)  to  our  error  occurrence  analysis,  error 
recovery  analysis,  skill  degradation  analysis,  and  other  data  analysis  topics  for  journal 
publications. 

•  Applied  our  developed  data  dictionary  for  use  by  all  current  and  future  research  team 
members  in  understanding  our  data  set  and  statistical  analyses  for  publication  writing. 

•  Abstract  submitted  to  the  International  Anesthesia  Research  Society  (IARS)  and  the 
Association  of  University  Anesthesiologists  (AUA)  2016  Meeting  “Critical  Errors  in 
Rarely  Performed  Procedures  0.5-5  Years  After  Training  Among  85  Surgeons”  (22 
January  2016)  (Appendix  53). 

•  Manuscript  submitted  to  the  Human  Factors  &  Ergonomics  Society  (HFES)  2016 
meeting  “Sensor-Free  Computer-Vision  Hand-Motion  Entropy  and  Video- Analysis  of 
Technical  Performance  during  Open  Surgery:  Proof  of  Concept  Report  of  Methodology 
and  Analysis”  (3  February  2016)  (Appendix  31). 


Future  Plans 

•  Submitted  pre -proposal  (September  2015)  (Appendix  39).  Invited  to  submit  full  proposal 
ASSET  3  Full  Proposal  BA150808  -  “Emergency  Refreshing  of  Combat  Surgical  Skills” 
(December  2015)  (Appendix  23).  Complete  full  proposal  by  due  date:  20  March  2016. 

•  Submitted  pre -proposal  “Autonomous  Analysis  for  Technical  Performance  of  Combat 
Surgical  Skills”  to  the  Office  of  Naval  Research  (30  September  2015).  Request  to  modify 
submission  (2  October  2015).  Submitted  modified  pre-proposal  (January  2016) 
(Appendix  37).  Awaiting  outcome  of  submission. 

•  Continuing  write-up  of  data  for  peer-reviewed  journal  publications. 


Conclusion 

This  project  progressed  extremely  well  and  was  on  target  for  all  Statement  of  Work  tasks. 
Institutional  Review  Board  approvals  were  obtained  swiftly  and  are  up  to  date.  Comparisons  of 
the  Phase  II  (before  and  after  training)  resident  skills  to  those  of  expert  surgeons  and  previously 
trained  surgeons  yielded  results  that  were  generally  supportive  of  the  study  hypotheses.  These 
results  also  indicated  that  the  metric  tool  developed  to  assess  skill  was  capable  of  discriminating 
skill  levels.  The  mobile  platform  and  cloud-based  evaluation  download  greatly  facilitated  the 


utility  of  the  metric.  However,  additional  adjustments  were  necessary  to  improve  efficiency. 
Preliminary  Analyses  of  Phase  1  and  Phase  11  video  task  analysis  indicated  that  there  was  good 
inter-rater  reliability  for  many  of  the  evaluation  criteria  for  distinguishing  expert  surgical 
technical  performance  and  confirmed  the  utility  of  remove  skills  evaluation.  Phase  2  studies 
compared  surgical  technical  skills  metrics  before  and  after  ASSET  training  and  38/40  Phase  2 
surgeons  completed  their  follow-up  12  to  18  months  after  their  ASSET  training.  Phase  3 
(previously  ASSET  trained)  collected  35  evaluations.  The  logistics  of  travel,  the  recent  winter 
weather  and  limitations  of  access  to  current  contact  information  impacted  the  process  of 
recruiting  this  group.  However,  analyses  of  the  complete  data  set  indicated  that  the  skills 
degradation  for  this  group  followed  expected  and  objectively  measurable  patterns.  This 
population  yields  fruitful  lines  of  inquiry.  Ten  experts  were  also  evaluated  and  these  data  greatly 
enhanced  comparisons  of  skill  and  objective  identification  of  competency  and  improvement. 

Discuss  according  to  Phase  II  data  (longitudinal  study):  No  skill  degradation  seen  for  whole 
cohort  in  vascular  exposure  and  control  procedures  (12-18  months  after  training);  a  priori  effects 
and  sample  size  were  met  since  95%  (38  out  of  40)  surgeons  returned  for  complete  follow-up, 
however,  there  was  a  wide  variety  of  surgeon  performance  amongst  the  cohort;  more  than  half  of 
the  surgeons  reached  within  1  nearest  neighbor  classifier  (equivalent  to  1  standard  deviation)  of 
the  expert  trauma  attending  surgeon  cohort;  remained  with  no  change  at  the  18  months  follow-up 
after  training;  critical  error  rates  were  high  pre -training  and  significantly  reduced  as  a  result  of  the 
ASSET  training;  specific  components  of  performance  that  were  significantly  improved  include 
anatomic  knowledge,  landmarks  and  skin  incision,  procedural  steps,  and  shortened  time  to 
successful  passage  and  double  vessel  looping;  error  recovery  also  improved  after  ASSET  course 
training 

Phase  111:  there  was  skill  degradation  but  we  don't  have  the  pre  data  to  compare  it  to;  1)  overall 
IPS  2)  issues  related  to  increase  in  critical  error  and  3)  error  recovery  was  significantly  different 

Fasciotomy:  sentinel  ASSET  procedure  discriminating  ASSET  Paining  Phase  III  surgeons  from 
others  after  training;  identifying  incomplete  fasciotomy  as  an  error  prone  outcome 

Experts:  4  of  the  10  failed  to  adequately  decompress  at  least  one  of  the  four  lower  extremity 
compartments 


Mitigation  plans 


Appendices 

Appendix  1:  Evaluation  of  Individual  Surgeon  Technical  Skills  during  Four 
Emergency  Procedures 

Colin  F  Mackenzie,  Evan  Garofalo,  Hegang  Chen,  Valerie  Shalin,  Kristy  Pugh,  Stacy 
Shackelford,  Sharon  Henry,  Mark  Bowyer,  Mark  Fitzgerald,  Joost  Funke  Kupper, 

George  Hagegeorge,  Peter  Hu,  Kon  Mouzakis. 

Background:  Maintaining  trauma  specific  surgical  skills  is  a  challenge  for  military 
surgeons.  Objective  assessment  of  surgical  readiness  is  needed.  We  hypothesized  that 
expert  surgeon  technical  skill  metrics  could  provide  a  reliable  technical  skill  assessment 
for  less  experienced  surgeons. 

Methods:  After  Institutional  Review  Board  approvals,  surgical  technical  skills 
assessment  metrics  were  developed  from  discussion  with  10  expert  surgeons,  video 
review  performing  three  vascular  exposure  procedures  and  lower  extremity  fasciotomy 
on  both  cadavers  and  hyper-realistic  physical  models,  and  a  consensus  conference.  These 
same  metrics  were  tested  in  10  residents  using  Android  tablet  software  and  a  head  camera 
to  capture  16  specific  steps  and  techniques  and  5  performance  global  ratings  during  the 
four  procedures.  Performance  was  then  assessed  on  random  video-clips  of  both  experts 
and  novices  by  5  trained  evaluators  and  compared  with  regression  modeling  and  inter¬ 
rater  reliability  (ICC)  analysis. 

Results:  Among  10  residents,  scores  showed  no  evidence  of  floor  or  ceiling  effects. 
Occurrence  of  16  expert  technical  skills,  agreed  upon  by  experts,  was  found  in  51-59%  of 
residents  during  the  4  procedures.  Global  overall  performance  rating  was  54%.  Global 
technical  performance  was  69%.  ICC  ranged  from  0.79-0.99  for  agreement  both  between 
raters  and  among  most  skills  ratings. 

Discussion:  Evaluation  metrics  discriminated  novices  from  an  expert  standard  with 
excellent  inter-rater  reliability.  Validation  in  a  larger  population  and  before/after  skills 
training  is  required.  Further  work  with  simulated  physical  models  may  provide  a  mobile 
skills  evaluation  platform  without  cadavers. 
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Appendix  2:  Development  of  a  Trauma  Readiness  Metric  Score  for  Surgeons 


Evan  Garofalo,  Stacy  Shackelford,  Valerie  Shalin,  Megan  Holmes,  Jason  Pasley,  Elliot 
Jessie,  Babak  Sarani,  Sharon  Henry,  Mark  Bowyer,  Colin  Mackenzie 

Background:  Maintaining  trauma -related  surgical  skills  during  peace-time  practice  is  a 
challenge.  Vascular  exposure  skill  in  trauma  is  an  essential  preparation  for  deployment. 
To  create  the  necessary  trauma- specific  skills  evaluation  protocol,  we  hypothesized  that 
comparison  of  expert  and  novice  surgical  knowledge  and  technical  skills  would  identify 
discriminatory  metrics  for  deployment  readiness. 

Methods:  Through  video  task- analysis  of  10  attending  trauma  surgeons  and  10  general 
surgery  residents  performing  three  vascular  exposures  [axillary  (AA),  brachial  (BA), 
femoral  arteries  (FA)]  and  lower  extremity  fasciotomy  (FAS)  in  fresh  cadavers,  we 
identified  knowledge  and  technical  skills  common  among  experienced  surgeons. 
Knowledge :  8  questions,  including  knowledge  of  injuries,  indications  for  surgery  and 
complications.  Technical,  completion  of  10  specific  surgical  steps  and  techniques,  6 
operative  maneuvers  common  to  experts,  and  5  performance  global  ratings  were 
combined  to  create  a  trauma  readiness  score.  These  were  assessed  for  10  residents  in  a 
blinded  video  review  by  5  trained  evaluators  with  item  analysis  and  descriptive 
regression  modeling  for  development  Trauma  Readiness  Index  (TRI). 

Results:  Inter-rater  reliability,  determined  by  intra-class  correlation  coefficient  ranged 
between  0.79-0.98.  Average  scores  were:  knowledge  questions  AA  62%,  BA  81%,  FA 
93%,  FAS  62%  answered  correctly;  expert  technical  skills  found  in  residents:  AA  55%, 
BA  51%,  FA  59%,  FAS  53%.  Global  ratings  (median  Fikert  1-5,  average  %  correct): 
Technical  Skills:  3,  54;  Indications/Complications:  3,  81;  Anatomy  knowledge:  3,  78; 
Readiness  2.5;  Global  evaluation  impression  score  (1-100)  69.  The  average  TRI  for  all 
procedures  was  62/100. 

Conclusion:  Trauma  readiness  metric  discriminated  expert  from  novice  performance  in 
both  technical  and  non-technical  skills  with  excellent  interrater  reliability.  Validation  in  a 
larger  population  and  with  trauma  skills  training  is  required. 
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Appendix  3:  Request  for  Budget  Modification  with  Justification  2014 


June  27, 2014 
Dear  Ms.  Bane. 

I  would  like  to  request  approval  of  some  budget  modifications  to  W81XWH-13-2-008 

“Use  of  Performance  Measures  to  Evaluate,  Document  Competence  and  Deterioration  of 
Advanced  Surgical  Skills  Exposure  for  Trauma  (ASSET)  Surgical  Skills".  The  Title  was 
abbreviated  as  Retention  and  Assessment  of  Surgical  Performance  (RASP) 

PRINCIPAL  INVESTIGATOR:  Dr  Colrn  Mackenzie 

These  budget  modifications  will  not  change  the  overall  cost  of  the  project,  nor  will  the}'  change 
the  scope  of  work  or  the  deliverables. 

The  justification  for  the  requested  budget  changes  is  shown  below  and  the  source  of  funds  is  the 
carry-over  of  S212K.  That  occurred  from  year  1:- 

The  carry-over  of  funds  that  occurred  m  Year  1  was  due  to  factors  shown  m  1)  below,  including 
slow  start-up  due  to: 

1)  an  inability  to  hire  qualified  personnel  candidates  with  expertise  m  Human  Anatomy  for  the 
Research  Associate  Assistant  positions  despite  advertising  in  the  Anatomy  journals,  newspaper 
and  within  the  UM  system.  2)  Non-hinng  of  a  Bio-informatics  person  as  we  originally  had  no 
data,  but  now  plan  to  hire  for  this  position  (see  7  below).  We  were  constrained  by  Sequestration 
for  our  Kick-Off  meeting  m  April  2013.  No  personnel  were  able  to  travel  from  Ft  Detiick,  but 
we  set  up  a  Telecommunication  line,  hi  addition  the  mid  -tenn  meeting  we  had  scheduled 
coincides  with  a  Mid  Term  reporting  invitation  by  Ft  Detiick.so  these  fund  expenditures  were 
less  than  budgeted  Dr.  Valerie  Shalin,  one  of  our  Consultants  was  on  Sabbatical  Leave  m 
France,  so  although  she  cbd  participate  in  our  weekly  conferences  by  Telecommunications  it  was 
not  for  as  many  hours  as  budgeted  She  is  now  back  from  Sabbatical  and  will  ‘catch  up’  these 
hours.  Travel  re-imbursement  requests  for  Phase  1  and  Phase  2  candidates  were  less  than 
budgeted  and  Evaluator  honorarium  were  not  all  paid  out  as  approval  has  not  been  obtained  (see 
item  3  and  9  below).  Our  video  technician  (George  Hagegeorge)  was  budgeted  at  50%  effort  but 
only  provided  30%  effort  during  year  1  start  up.  Now  that  we  are  collecting  and  rendering  hugely 
mcreased  amounts  of  video  data  we  would  like  approval  to  increase  his  effort  to  70%  as  of  July 
1“  2014 

We  request  modification  of  the  budget  to  use  the  year  1  carry-over  funds  as  follows: 

1)  Increase  in  expenditure  on  Faculty  Salary:  Faculty  Salary'  costs  have  mcreased  beyond 
budget  because  Dr.  Hegang  Chen  (Statistician  to  the  effort)  has  been  promoted  from 
Associate  to  Full  Professor  in  the  Department  of  Epidemiology  (budgeted+15%: 
$130,002)  and  Dr.  Hu  has  been  promoted  from  Assistant  to  Associate  Professor  in  the 


Appendix  4:  Application  Training  Module 
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Appendix  5:  Mobile  Platform  to  Evaluate  Individual  Surgeon  Technical  Skills 
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Background:  We  developed  surgical  technical  skills 
assessment  metrics  based  on  discussion  with  expert 
surgeons,  video  review  of  them  performing  four  surgical 
procedures  and  a  consensus  conference.  We  describe  the 
utility  of  a  mobile  app  and  video/audio  capture  system  to 
gather  data  on  surgical  technical  skills  useful  for  training, 
telemedicine  applications,  evaluation  in  real  surgery, 
cadavers  and  simulated  surgical  models. 

Methods:  Software  was  developed  for  Android  tablets  to 
capture  assessments,  which  were  uploaded  to  the  cloud  for 
analysis  by  trained  evaluators  (Fig  1).  The  system  was  tested 
during  axillary,  brachial,  femoral  artery  exposure,  and  lower 
extremity  fasciotomy  surgery.  For  video/audio  capture  we 
compared  three  camera  systems:  a)  Pan-tilt-zoom  [PTZ] 
ceiling  mounted,  b)  on  mobile  stand,  c)  head-mounted 
camera  with  laser  pointer  (Figs  2-4)  and  three  audio  capture 
systems:  a)  head  worn  boom  microphone  (Fig  5),  b)  ceiling 
mounted  and,  c)  audio  capture  with  head  camera). 

Results:  The  tablet  facilitates  capture  of  surgeon  evaluation 
metrics  less  intrusively  and  without  paper  copies.  Data  from 
multiple  evaluations  stored  in  the  cloud  avoids  data 
transcription.  The  $120  head-  mounted  laser-directed 
camera  captured  up  to  1.5  hours  of  surgical  video  and  audio 
adequately  to  assess  performance  metrics  remotely.  The 
PTZ  camera  with  boom/overhead  audio  system  ($90K)  gave 
the  best  image,  but  was  obstructed  by  the  surgeon's  head. 
The  mobile  stand  mounted  camera  required  constant 
movement  to  capture  images  of  the  surgical  field. 
Discussion:  An  inexpensive  mobile  data  and  video/audio 
capture  system  could  be  used  to  non-intrusively  collect  data 
to  evaluate  surgeon  technical  performance,  with  remotely 
situated  evaluators.  Further  work  will  test  simulated 
physical  models  (Fig  6)  of  the  four  surgical  procedures  with 
enough  fidelity  to  realistically  challenge  surgical  technique, 
to  provide  a  mobile  surgical  skills  evaluation  platform  ready 
for  pre-deployment  testing  without  the  constraints  of 
cadaver. 


Figure  4  Figure  5 

Focusing  the  Head  Cam  Head  Cam  and  boom  microphone 
laser  pointer 


Figure  6 

Hyper-Realistic  Physical  Model  for  Fasciotomy 


Figure  1 

Tablet  Showing  RASP  App, 
with  Script,  Case  History, 
and  Evaluation  metrics 


Figure  3 

RASP  evaluation  showing  participant  wearing  head  camera 


Figure  2 

Detail  of  Head  Cam  and  Laser 


Approved  for  release .  U.S.  Army  Medical  Research  S  Materiel  Command  (USAMRMC)  and  the  Telemedicine  &  Advanced  Technology  Research  Center  (TATRC),  at  Fort  Detrick,  Md  under  Contract  Number:  W81XWH-1 3-2-0028 

The  views,  opinions  and/or  findings  contained  in  this  presentation  are  those  of  the  author(s)  and  do  not  necessarily  reflect  the  views  of  the  Department  of  Defense  and  should  not  be  construed  as  an  official  DoO/Army  position  of  policy 


Appendix  6:  Physical  Model  Realism  Questionnaire 


Upper  Extremity  Model  Realism  Feedback 

Please  score  the  realism  of  Upper  Extremity  Model  features  below  on  a  scale  of  1  to  5 


Skin 

Subcutaneous  tissue 

Muscle 

Fascia 

Vasculature 

Usefulness  for  Training 

Realism  for  training 

Anatomic  reality 


1=  No  reality  5  =  Very  realistic 
1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 


For  the  Upper  Extremity  model,  please  protide  feedback  on  the  following: 

What  are  the  strengths  of  the  model? 


What  are  the  weaknesses? 


Did  you  find  anything  about  the  model  distracting? 


Do  you  have  suggestions  for  improvement? 


Any  other  comments? 


Thank  you  for  your  participation ! 


Appendix  7:  Cadaver  vs.  Live  Patient  Questionnaire 


Cadaver  Upper  Extremity  Realism  Feedback 

Compared  to  a  live  patient,  please  score  the  cadaver  upper  extremity  on  a  scale  of  1  to  5 


Skin 

Subcutaneous  tissue 

Muscle 

Fascia 

Vasculature 

Usefulness  for  Training 

Realism  for  training 

Anatomic  reality 


1=  No  reality  5  =  Very  realistic 
1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 


For  the  cadaver  upper  extremity,  please  provide  feedback  on  the  following: 

What  are  the  strengths  of  the  model? 


What  are  the  weaknesses? 


Did  you  find  anything  about  the  model  distracting? 


Do  you  have  suggestions  for  improvement? 


Any  other  comments? 


Thank  you  for  your  participation ! 


Appendix  8:  Study  Participant  Information  and  Confidence  Questionnaire 


RASP  Study  Participant  Information 


Demographic  Information 

Name _  Age _ Sex _ 

Institution _ Clinical  years _ 

Status  (circle  one):  Resident  Chief  Resident  Fellow  (PGY-6 _  PGY-7  )  Attending 

Address _ 

Email _  Phone _ 

Surgical  Experience 

What  is  your  surgical  (sub)  specialty? _ 

Number  of  months  on: 

Trauma  Service _  non-trauma  Acute  Care  Service _ 

Please  estimate  the  time  since  you  last  performed  surgery:  Years _  Months  Days _ 

Please  give  the  approximate  number  of  patients  for  each  of  the  following: 

Trauma  patients  you  have  treated  or  evaluated _ 

Percentage  of  trauma  patients  with  penetrating  trauma _ % 

Estimate  the  number  of  trauma-related  procedures  you  have  participated  in  for  the  following: 

1.  Upper  extremity  vascular  repairs  (open)  _ 

2.  Upper  extremity  vascular  repairs  (endovascular)  _ 

3.  Lower  extremity  vascular  repairs  (open)  _ 

4.  Lower  extremity  vascular  repairs  (endovascular)  _ 

5.  Lower  extremity  fasciotomy  _ 

Estimate  the  number  of  non-trauma  related  procedures  you  have  participated  in  for  the  following: 

1.  Upper  extremity  vascular  procedures  for  dialysis 

access  _ 

2.  Other  upper  extremity  non-dialysis  vascular 

procedures  - 

3.  Lower  extremity  open  vascular  procedures  - 

4.  Lower  extremity  endovascular  procedures  - 

5.  Lower  extremity  fasciotomy  - 

Other  than  anatomy  laboratory  during  medical  school,  please  estimate  the  number  of  hours  you  have  spent  in  a  cadaver 
laboratory: _ 

Have  you  taken  any  cadaver  based  courses  since  medical  school?  _ Yes  _ No 

If  yes,  please  specify: _ 

Estimate  the  amount  of  time  you  have  spent  in  a  skills  laboratory  during  your  training  or  in  other  activities: 

Minimally  Invasive  skills  tasks: _ hours 

Open  operative  skills  tasks: _ hours 

ASSET  course  date _  Participant  ID _ 


Appendix  9:  Management  of  Vascular  Trauma  by  Senior  Surgical  Residents:  Perception 
Does  Not  Equal  Reality 
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Introduction:  Experience  with  the  management  of  vascular  trauma  by  senior 
surgical  residents  is  imrted  When  queried  about  their  understanding  of  anatomy 
and  ablity  to  perform  specific  vascular  exposures,  residents  express  a  moderately 
high  level  of  confidence.  We  hypothesized  that  this  perception  does  not  equal 
reality. 

Methods:  42  senior  scrgical  residents  participating  in  an  ongoing  validation  study  of 
the  Advanced  Surgical  Skills  for  Exposures  in  Trauma  (ASSET)  course  were  asked 
to  seif-assess  their  baseline  (pre-course)  confidence  of  their  understanding  of  the 
anatomy  required  to  perform,  and  their  ability  to  perform  exposures  of  the  Axillary 
(AA),  Brachial  (BA),  and  Femoral  (FA)  Arteries,  as  well  as  Lower 
Extremity  Fasciotomy  (LEF)  using  a  5  point  Likert  scale.  The  residents  then 
performed  the  4  procedures  on  a  cadaver  model  and  were  scored  in  real  time  by 
pre -trained  trauma  experts  using  both  a  global  assessment  (5  point  Likert  scale)  of 
’understanding  of  anatomy'  and  "resident  is  ready  to  perform",  as  well  an  overall 
numerical  score  (1-100)  of  the  performance.  Statistical  analysis  was  performed 


Appendix  10:  Development  and  Validation  of  Trauma  Surgical  Skills  Metrics:  Preliminary 
Assessment  of  Performance  after  Training 


EAST  2015  Plenary  Paper 


Development  and  validation  of  trauma  surgical  skills  metrics: 
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Trauma  surgical  training  remains  a  core  competency  of 
general  swgcry  residencies;  however,  opportunities  to  tain 
and  develop  trauma-specific  surgical  skills  hjyc  decreased  dur- 
ing  the  past  decade.  A  number  offartorshavcconfcinedtoroduce 
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the  total  operative  trauma  cases  for  graduating  chief  residents, 
including  a  gradual  shift  in  management  stalcgjcs  for  traumatic 
injuries  witi  inocasod  noniape  rathe  management,1  ~J  a  in  ft  in 
the  epidemiology  of  taumatic  injury  with  reduced  penetrat¬ 
ing  trauma  nationwide,4  and  implcnhintation  of  work  hour  re¬ 
strictions  since  2003. 

Simultaneously,  advances  in  simulation  training  have 
occurred,  offemg  a  potential  nvxhanion  to  supplement  nirgkal 
skills  training.  Simulation  training  options  include  virtual- 
reality  trainers,’-10  models  or  manikins,11-13  human  cadaver 
training,14-*  and  animal  laboratories.17-20  A  number  of  sur¬ 
gical  skills  assessmert  method*  have  been  developed  during  the 
past  decade,  which  include  foe  general  categories  of  task-specific 
checklists,  global  rating  scales,  procedure-specific  rating  scales, 
and  nonfcchnical  &iDs  assessments. 21-23  An  objective  method  of 
assessing  competency  in  sugical  ski  Os  specific  t>  trauma  siigety 
would  be  very  beneficial  to  surgical  Raining  programs.  Military 
surgeons  are  faced  with  the  additional  challenge  of  maintaining 
trauma-specific  surgical  skills  necessary  for  wartime  deployment. 
Roitinc  trauma  care  is  not  provided  in  the  da  fly  practice  of  most 
mOitvy  surgeons,24  so  that  interval  assessment  of  trauma- 
specific  surgical  skills  ever  time  is  needed  to  ensure  that  previ¬ 
ously  Rained  surgeons  remain  prepared  for  deployment 
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Appendix  11:  Mobile  Platform  to  Evaluate  Individual  Technical  Skills 
American  Society  of  Anesthesiologists,  October  2014 


Colin  F  Mackenzie,  Mark  Fitzgerald,  Kon  Mouzakis,  Joost  Funke  Kupper,  George 
Hagegeorge,  Peter  Hu,  Evan  Garofalo,  Mary  Njoku,  Stacy  Shackelford 

Background:  Many  medical  specialty  Boards  have  proposals  to  include  a  simulated 
patient  encounter  to  test  technical  skills  in  addition  to  knowledge-based  oral 
examinations  in  future  certification  processes.  We  developed  surgical  technical  skills 
assessment  metrics  based  on  discussion  with  expert  surgeons,  video  review  of  10  experts 
performing  four  procedures,  and  a  consensus  conference.  We  describe  the  utility  of  a 
mobile  app  and  video/audio  capture  system  that  was  used  to  gather  data  and  results  of 
testing  such  surgical  technical  skills  metrics,  as  the  principles  have  widespread 
applicability  for  evaluation  of  anesthesiology  training  and  real-time  clinical  management, 
as  well  as  for  telemedicine  and  research  applications. 

Methods:  After  Institutional  Review  Board  approvals,  an  audio- video  data  capture 
system  was  tested  during  four  surgical  procedures  performed  on  both  cadavers  and  hyper- 
realistic  physical  models  by  10  residents.  Software  was  developed  for  Android  tablets  to 
capture  assessments  of  knowledge  (indications,  complications)  and  technical  skills  (10 
specific  steps  and  techniques,  6  technical  maneuvers  and  5  performance  global  ratings) 
common  to  the  10  experts.  Results  were  uploaded  to  the  cloud  for  blinded  video  analysis 
by  5  trained  evaluators  and  tested  with  regression  modeling  and  inter-rater  reliability 
(ICC)  analysis.  We  also  tested  three  camera  systems  (Pan-tilt-zoom  [PTZ]  ceiling 
mounted,  on  mobile  stand,  head-mounted  camera  with  laser  pointer)  and  three  audio 
capture  systems  (head  worn  boom  microphone,  ceiling  mounted  and  audio  capture  with 
head  camera). 

Results:  The  tablet  facilitates  capture  of  surgeon  evaluation  metrics  with  little  intrusion 
and  without  paper  copies.  Data  from  multiple  evaluations  stored  in  the  cloud  avoids  data 
transcription.  The  $120  head-mounted  laser-directed  camera  captured  1.5  hours  of 
surgical  video  and  audio  adequately  to  assess  performance  metrics  remotely.  The  PTZ 
camera  with  boom/overhead  audio  system  ($90K)  gave  the  best  image,  but  was 
obstructed  by  the  operator’s  head.  The  mobile  stand  mounted  camera  required  constant 
movement  to  capture  images  of  the  surgical  field.  Inter-rater  reliability  (ICC)  among 
technical  skills  assessments  ranged  between  0.79-0.98.  Among  10  residents,  knowledge 
questions:  62-  93%  were  answered  correctly.  Occurrence  of  16  expert  technical  skills  was 
found  in  51-59%  of  residents.  Global  ratings  (median  Likert  1-5  and  overall 
%);Technical:  3  overall  54%;  Indications/Complications  3  with  81%  correct;  Anatomy 
knowledge  3  with  78%  correct,  Global  Evaluation  69%. 


Discussion:  Using  an  inexpensive  mobile  data  and  video/audio  capture  system  with 
remotely  situated  evaluators,  performance  metrics  discriminated  experts  from  novices  for 
technical  and  non-technical  skills  with  excellent  inter-rater  reliability.  Validation  in  a 


larger  population  and  before/after  skills  training  is  required.  Further  work  will  test  the 
simulated  physical  models  of  the  four  procedures  with  enough  fidelity  to  realistically 
challenge  technique,  to  provide  a  mobile  skills  evaluation  platform.  Funded  by 
W81XWH- 13-2-0028. 


Appendix  12:  Accurate  Assessment  of  Surgical  Skill  Improvements  after  Training 

Presentation:  Eastern  Association  of  Surgery  for  Trauma  (EAST,  January  2015) 

Stacy  Shackelford,  MD,  Evan  Garofalo,  PhD,  Valerie  Shalin,  PhD,  Kristy  Pugh,  MS, 
Jason  Pasley,  DO,  Babak  Sarani,  MD,  Sharon  Henry,  MD,  Mark  Bowyer,  MD,  Colin 
Mackenzie  MBChB 

Background:  Maintaining  trauma  specific  surgical  skills  is  an  ongoing  challenge  for 
surgical  training  programs.  An  objective  assessment  of  surgical  skills  is  needed.  We 
hypothesized  that  a  reliable  surgical  skills  assessment  tool  could  detect  knowledge  and 
skill  differences  following  a  training  intervention. 

Methods:  After  Institutional  Review  Board  approval,  we  developed  surgical  technical 
skills  assessment  metrics  based  on  discussion  with  expert  surgeons,  video  review  of  10 
experts  performing  four  vascular  exposure  procedures  on  both  cadavers  and  hyper- 
realistic  physical  models,  and  a  consensus  conference.  We  then  tested  knowledge  and 
technical  skill  metrics  in  12  surgical  residents  (year  3-5)  before  and  2  weeks  after 
vascular  exposure  skills  training  with  the  Advanced  Surgical  Skills  for  Exposure  in 
Trauma  course.  Performance  was  assessed  by  six  trained  evaluators;  data  was  recorded 
using  Android  tablet  software  and  a  head  camera  to  capture  technical  skill  assessments. 
Performance  was  assessed  in  three  areas:  knowledge  (anatomic,  indications, 
management),  procedural  steps,  and  technical  skills.  Time  to  completion  of  procedures 
was  recorded. 

Performance  scores  were  calculated  before  and  after  training.  Wilcoxon  paired  t  was  used 
to  examine  statistical  significance  at  alpha<  0.05. 

Results:  Trauma  Readiness  Index  for  three  vascular  exposures  and  lower  extremity 
fasciotomy  improved  by  14%  after  training.  Sorted  by  specific  skills,  the  skill  most 
improved  by  1-day  skills  was  procedural  steps,  scores  increased  20%.  Technical  skill 
scores  improved  12%.  Overall  knowledge  improved  3%,  with  further  analysis  localizing 
this  effect  to  a  17%  improvement  in  anatomic  knowledge.  Time  to  complete  procedures 
decreased  4.3  minutes  (13.4  to  9.1  min). 

Conclusion:  A  detailed  surgical  skills  assessment  is  a  valuable  tool  to  assess  a  variety  of 
surgical  training  programs.  The  measurement  tool  detected  improvements  in  specific 
procedural  steps  and  anatomic  knowledge  taught  during  a  1-day  course.  The  tool  also 
detected  improvements  in  technical  skills  and  management  normally  acquired  during  the 
course  of  residency  training.  Future  applications  will  include  assessing  specific  skills 
acquired  during  the  course  of  residency  training. 


Surgical  Skills  Assessment  Scores 


Pre-training 

Post-training 

Mean 

Std 

Mean 

Std 

Improvement 

P-value 

Dev 

Dev 

Knowledge  score* 

•  Overall 

50 

13 

53 

14 

3 

0.013 

•  Anatomic 

50 

16 

50 

16 

17 

0.00001 

•  Management 

43 

17 

45 

15 

2 

0.044 

Technical  skills  score* 

59 

18 

71 

17 

12 

0.0001 

Procedure  steps  score* 

46 

23 

67 

16 

20 

0.0000001 

Time  (minutes) 

13.4 

5.9 

9.1 

4.5 

-4.3 

0.000001 

Trauma  Readiness  Index* 

50 

12 

64 

10 

14 

0.0001 

*Scores  represent  the  percentage  of  expert  surgeon  performance  skills  found  in  residents 
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Appendix  13:  Assessment  of  Surgical  Anatomy  Skills  in  Upper  and  Lower  Limb  Vascular 
Control  before  and  after  Training 

Podium  presentation:  Association  for  Surgical  Education  (April  2015) 

Evan  Garofalo,  PhD,  Stacy  Shackelford,  MD,  Valerie  Shalin,  PhD,  Kristy  Pugh, 

MS,  Hegang  Chen,  PhD,  Jason  Pasley,  DO,  Babak  Sarani,  MD,  Sharon  Henry,  MD, 
Mark  Bowyer,  MD,  Colin  Mackenzie  MBChB 

Background:  Maintaining  trauma  specific  surgical  skills  is  a  challenge  for  military  and 
civilian  surgeons.  We  hypothesize  that  a  trauma  training  course  including  rapid  upper  and 
lower  limb  vascular  exposure  improves  correct  identification  of  surgical  landmarks, 
anatomical  structures,  and  shortens  time  to  vascular  control.  Specifically,  improved 
knowledge  of  surface  landmarks  is  associated  with  faster  procedures  and  more  successful 
vessel  identification  and  control. 

Methods:  We  developed  a  surgical  skills  evaluation  tool  through  discussion  with  expert 
trauma  surgeons,  video  review  of  10  experts  performing  three  open  vascular  exposures 
(axillary  [AA],  brachial  [BA]  and  femoral  arteries  [FA])  on  cadavers,  and  a  consensus 
conference.  An  Android  application  was  designed  to  run  the  tool  and  two  trained 
evaluators  assessed  the  technical  skills  of  34  surgical  residents  (years  3-5)  while 
performing  these  procedures  before  and  after  completing  the  Advanced  Surgical  Skills 
for  Exposure  in  Trauma  (ASSET)  course.  Correct  identification  of  surface  anatomical 
landmarks,  incisions,  structures  in  surgical  procedural  steps,  and  time  to  completion  of 
procedure  were  compared  before  and  after  training  using  Pearson  Correlation  and  Linear 
Mixed  Models  test. 

Results:  Table  1  details  results  of  the  analyses.  In  AA  and  FA  procedures,  there  is  a 
significant  effect  of  ASSET  training  showing  decreased  procedure  time  (p<0. 00 1 ), 
improved  surface  landmark  identification  (pcO.OOl:  AA  40%;  BA  15%;  FA  24%), 
correct  vessel  identification  and  vessel  control  (AA,  FA  pcO.OOl).  There  was  significant 
correlation  between  correct  surface  landmarks,  incisions  and  artery  identification  with 
successful  vascular  control  for  AA  and  FA  (r=0.25  to  0.44;  all  correlations  pcO.OOl). 
Decreased  procedure  time  was  correlated  with  correct  surface  landmarks  (pc 0.02)  and 
incisions  (pcO.OOl)  for  the  BA  procedure.  Neither  residency  year  nor  evaluator  had  an 
effect  on  the  pre  and  post  scores. 

Conclusions:  Documentation  of  correct  surface  landmarks  and  incisions  was  associated 
with  swift  successful  control  of  upper/lower  limb  vasculature.  By  this  measure,  structural 
recognition  during  specific  procedural  steps  and  surface  anatomic  knowledge  were  highly 
impactful  information  taught  during  a  1-day  course.  This  training  approach,  normally 
acquired  during  the  entirety  of  residency  training,  may  help  accelerate  acquisition  of 
emergency  surgery  specific  skills  to  compensate  for  shortened  training  hours  or  when 
just-in-time  training  is  necessary. 


Table  1:  Comparison  of  anatomical  knowledge  and  surgical  performance  between  Pre  and 
Post  training  for  Axillary,  Brachial  and  Femoral  artery  exposures  using  Linear  Mixed  Models 

Procedure 

Axillary  Artery  Brachial  Artery  Femora  I  Artery 


mean  ±  SD 

mea  n  ±  SD 

mean  ±  SD 

mean  ±  SD 

mean  ±  SD 

Surface 

Pre 

35  ±4 

49  ±3 

56  ±3 

Landmarks 

Post 

74  ±4 

65  ±3 

81  ±3 

(%  correct) 

diff 

^g  *** 

15  ** 

24  *** 

Adequate 

Pre 

-1.2  ±0.3 

1.1  ±0.4 

-0.7  ±0.3 

Incision  ¥ 

Post 

1.6  ±0.3 

2.9  ±0.6 

1.2  ±0.3 

diff 

2  g  *** 

0.6  ** 

^  g  *** 

Procedure 

Pre 

11.7  ±0.6 

10.4  ±0.7 

16.6  ±0.7 

Time  (mins) 

Post 

7.9  ±0.6 

9.5  ±0.7 

13.6  ±0.7 

diff 

-3  8  *** 

-0.8 

_2  g  *** 

CFA+ 

SFA+ 

PFA+ 

Correct 

Pre 

-0.2  ±0.3 

0.15  ±0.4 

-0.7  ±0.4 

0.2  ±0.4 

-0.4  ±0.3 

Artery 

Post 

2.3  ±0.5 

0.9  ±0.4 

1.6  ±0.4 

3.5  ±0.07 

1.8  ±0.3 

Identificatio 

diff 

2  g  *** 

0.8 

2  ^  *** 

^  4  *** 

2  2  *** 

Successful 

Pre 

-0.7  ±0.3 

0.2  ±0.4 

-0.4  ±0.3 

-0.1  ±0.4 

-0.6  ±0.4 

Artery 

Post 

1.9  ±0.4 

1.3  ±0.4 

1.8  ±0.4 

3.0  ±0.6 

1.4  ±0.4 

Control  ¥ 

diff 

2  g  *** 

1  * 

2  ^  *** 

^  *** 

2  *** 

+CFA:  Common  Femoral  Artery;  SFA:  Superficial  Femoral  Artery;  PFA:  Profunda  Femoral  Artery; 
¥  Binary  data  analysed  with  General  Linear  Mixed  Model,  logitistic  model 
Significance  of  difference  demarked  as  a=0.05:  ***p<  0.0001;  **  p<  0.001;  *p<  0.05 
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Appendix  14:  Mobile  Platform  for  Assessing  Emergency  Trauma  Surgical  Skill 
Performance 

Accepted  for  poster  presentation:  Association  for  Surgical  Education  (April  2015) 

Colin  Mackenzie,  Stacy  Shackelford,  Evan  Garofalo,  Hegang  Chen,  Jason  Pasley, 
Sharon  Henry,  George  Hagegeorge,  Kristy  Pugh,  Mark  Bowyer. 

Background:  Surgical  resident’s  operative  trauma  experience  has  decreased  from  60-35 
cases  1999-2012.  A  mobile  platform  would  be  useful  for  residency  programs  to  evaluate 
competence  in  trauma  skills.  We  tested  the  hypothesis  that  remote  review  of  video  clips 
discriminated  pre  from  post  training  performance  in  vascular  control  no  differently  than 
co-located  performance  evaluation. 

Methods:  Performance  of  surgical  skills  were  evaluated  by  two  co-located  trained 
experts  during  three  video-recorded  vascular  exposure  procedures  (Brachial  artery  (BA), 
Axillary  Artery  (AA),  and  Femoral  Artery  (FA))  performed  on  fresh  cadavers  by  ten  3  - 
5th  year  surgical  residents  before  and  within  2  weeks  of  Advanced  Surgical  Skills  for 
Exposure  in  Trauma  (ASSET)  training.  Metrics  included  landmarks,  specific  steps  and 
techniques,  expert  discriminators  and  global  performance  ratings  common  to  10  experts. 
The  performance  metrics  were  previously  validated  for  BA,  AA,  and  FA  with  inter  rater 
reliability  (ICC)  analysis  showing  ICC  0.7-  0.98  among  5  raters.  In  this  study,  Pre/Post 
training  video  clips  of  10  residents  for  each  procedure  were  randomly  ordered  for  blinded 
analysis  by  2  trained  evaluators  and  compared  to  same -procedures  simultaneously 
assessed  by  co-located  evaluations.  Co-located  and  video  evaluations  were  compared 
with  Pearson  Correlation  and  Linear  Mixed  Models. 


Results:  Evaluation  metrics  showed  no  floor  or  ceiling  effects.  Expert  discriminators 
(skin  incision,  logical  sequence,  anatomic  knowledge  etc.),  procedural  steps  (correct 
structure  identification)  and  global  ratings  (1=  poor  -  5=excellent)  of  anatomy  were  no 
different  among  co-located  and  video  evaluators  for  BA,  AA,  FA.  Differences  in  other 
global  ratings  of  skills,  readiness  and  overall  grade  (%)  were  variable  between  video  and 
co-located  evaluations  (Table). 

Discussion:  Remotely  situated  video  review,  had  agreement  in  objective  pre/post  training 
performance  with  co-located  evaluators,  but  not  in  more  subjective  assessments.  Video 
focused  on  the  surgeon’s  hands  could  account  for  these  differences.  Video  recordings  of 
cadaveric  vascular  exposure,  with  remote  evaluations  of  objective  metrics  assess 
residents’  vascular  exposure  competence. 


Expert 


Landmarks 

Technical  Points 

Discrimination 

Procedural 

Global 

Global  Tech 

Global 

(%) 

(°/o) 

(%) 

(°/o) 

Anatomy 

Skill 

Readiness 

Grade  (%) 

mean  ±  SD 

mean  ±  SD 

mean  ±  SD 

mean  ±  SD 

mean  ±  SD 

mean  ±  SD 

mean  ±  SD 

mean  ±  SD 

Pre 

86  ±6 

57  ±3 

55  ±5 

38  ±5 

2  ±0.2 

3  ±0.2 

2.2  ±0.2 

65  ±2.3 

Post 

46  ±6 

73  ±3.5 

75  ±5 

85  ±5 

3.5  ±0.2 

3.5  ±0.2 

3.7  ±0.3 

81  ±2.4 

Axillary 

Post  training 
Delta 

40  *** 

16  *** 

2o  *** 

47*  ** 

2  4*** 

2  *** 

2  5*** 

16  *** 

Co-located  vs. 

Video 

0.8 

0.02* 

0.59 

0.1 

0.09 

0.02* 

0.3 

0.1 

(p  value) 

Pre 

60  ±6 

62  ±3 

61  ±5 

62  ±5 

2.4  ±0.2 

2.8  ±0.2 

2.5  ±0.2 

69  ±3 

Post 

66  ±6 

68  ±3 

65  ±5 

79  ±6 

3.1  ±0.2 

3.1  ±0.2 

3.0  ±0.2 

75  ±3 

Brachial 

Post  training 
Delta 

7 

7 

5 

1.8*** 

0.8  *** 

0.3 

0.5* 

5.5  * 

Co-located  vs. 

Video 

0.75 

0.02* 

0.02* 

0.46 

0.15 

0.27 

0.1 

0.1 

(p  value) 

Pre 

70  ±3.9 

56  ±3.6 

37  ±5 

44  ±6.3 

2.5  ±0.2 

2.4  ±0.2 

2.2  ±0.2 

66  ±2.3 

Post 

85  ±3.9 

70  ±3.6 

70  ±5 

87  ±6.3 

3.5  ±0.2 

3.2  ±0.2 

3.4  ±0.2 

79  ±2.3 

Femoral 

Post  training 
Delta 

14* 

14  *** 

33  *** 

44.5  *** 

2*** 

q  g  *** 

2  2  *** 

23  *** 

Co-located  vs. 

Video 

0.004  *** 

0.04* 

0.5 

0.7 

0.2 

0.07 

0.04* 

0.004  *** 

_ ( p  value) _ 

a =0.05.  Significance  is  demarcated  as:  ***  p<  0.0001;  **  p<  0.001;  *  p<  0.05 


Table:  Shows  mean  ±  standard  deviation  (SD)  of  metrics  across  top  X  axis.  Along  the  Y 
axisare  shownAxillary,  Brachial  and  Femoral  artery  vascular  exposure  and  control 
procedures  performed  by  10  surgical  residents  with  the  Pre  Post  training  differences  and 
whether  the  evaluators  used  video  or  were  co-located.  Expert,  and  Procedural  evaluations 
and  Anatomy  Global  Ratings  were  not  different  between  video  and  co-located  evaluators. 


Appendix  15:  Surface  Anatomy  in  the  Performance  of  a  Lower  Extremity  Fasciotomy 
before  and  after  Training 

Accepted  for  poster  presentation:  Federated  American  Societies  for  Experimental 
Biology,  American  Association  of  Anatomists  (FASEB,  AAA  March  2015) 

Evan  Garofalo,  Stacy  Shackelford,  Valerie  Shalin,  Kristy  Pugh,  Hegang  Chen, 

Jason  Pasley,  Babak  Sarani,  Sharon  Henry,  Mark  Bowyer,  Colin  Mackenzie 

With  shorter  training  hours,  acquiring  trauma  surgical  skills  on-the-job  is  challenging  for 
civilian  and  military  surgeons.  We  hypothesize  that  a  training  course  including  lower 
extremity  (LE)  fasciotomy  will  improve  knowledge  of  surgical  landmarks,  anatomical 
structures,  and  procedure  time.  Specifically,  improved  knowledge  of  surface  landmarks 
will  correlate  with  faster  and  successful  4  compartment  decompression. 

Surgical  residents  (n=34)  were  tested  with  validated  metrics  performing  a  2  incision  4 
compartment  fasciotomy  on  a  cadaver  before  and  after  the  Advanced  Surgical  Skills  for 
Exposure  in  Trauma  (ASSET)  course.  Surface  landmarks,  incision  placement,  surgical 
procedural  steps,  and  procedure  time  were  compared  before  and  after  training  with  Linear 
Mixed  Models  and  Pearson  Correlation. 

After  training,  residents  improved  in  landmark  identification  (+33%),  incision  placement 
(+34%),  and  successful  4  compartment  decompression  (all  p<0.001).  More  compartments 
were  completely  opened  in  less  time  (ANCOVA  /;<(). 05;  figure).  Correct  landmarks  and 
incisions  correlated  with  successful  decompression  (r=0.42-0.5;  p<0.001). 

Improved  surface  anatomic  knowledge  increased  successful  fasciotomy.  This  knowledge 
is  normally  acquired  on  the  job  during  residency  but  specific  training  may  help  accelerate 
the  acquisition  of  fasciotomy  skills  to  compensate  for  reduced  training  hours. 


Anatomy  in  the  performance  of  a  lower  extremity  fasciotomy  before  and  after  training 
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Purpose 

Acquiring  and  maintaining  surgical  skills  for  trauma  has  become 
challenging  for  military  &  civilian  surgeons  due  to  mandated  shorter 
training  hours  (fewer  surgical  cases  observed),  more  non-operative 
management  for  trauma,  and  infrequent  encounters  with  specific 
procedures  and  injuries.  However,  skill-specific  training  programs  can 
supplement  on-the-job  training  and  are  used  to  maintain  trauma 
surgical  skills  for  deploying  surgeons  treating  combat-related  injuries. 
For  example  the  Advanced  Surgical  Skills  for  Exposure  in  Trauma 
(ASSET)  course  is  a  1  day  case-based  program  using  cadavers. 


Training  improved  surgical  performance  for  all  variables  but  time  (+30  secs)  (Table  1,  Fig  4) 

♦28%  surface  landmark  identification  after  training 
+32%  Incision  placement  after  training 

+  compartment  decompression  (pre-training=  1;  post-training=3.1) 

Correct  anatomical  structural  identification  is  correlated  with  more  complete  procedures,  more  successful  compartmental 
decompression,  and  higher  IPS  (Table  2) 

Correct  landmark  identification  and  incision  placement  correlated  with  successful  decompression  (Table  2,  Fig  5) 

Residents  decompressed  more  compartments  in  a  similar  amount  of  time  after  training  (Table  2;  Fig  6) 


One  procedure  that  most  surgeons  have  little  exposure  to  is  a 
fasciotomy  (for  trauma')  for  compartment  syndrome  of  the  leg  - 
caused  by  increased  pressure  from  swelling  or  external  pressure 
within  defined  space.  This  restricts  volume  and,  when  uncorrected  or 
incomplete,  can  result  in  loss  of  limb  from  nerve  and  tissue  necrosis  or 
possible  death.  During  conflicts  in  Iraq  and  Afghanistan,  17%  of 
fasciotomies  performed  in  the  field  were  incomplete2.  We  suspect 
that  familiarity  with  the  surgical  anatomy  improves  successful 
performance. 


H,:  knowledge  of  surgical  landmarks,  anatomical  structures  are 
associated  with  faster  and  more  successful  4  compartment 
decompression 


Methods 

•  Developed  fit  validated  objective  performance  tool  and  metrics 
(Fig  1  fit  2)  using  a  case-based  scenario  evaluation  platform 

•  Assessed  knowledge  and  technical  skills  by  checklist  for 
components:  Surface  landmarks,  incision  placement,  structure 
identification,  surgical  procedure  steps,  management,  technical 
skills  (Fig  3) 

-  Scores  calculated  as  points  awarded/points  possible  and 

combined  into  a  composite  score  Individual  procedure  score  (IPS) 

•  Surgical  residents  (n=34) 

•  Evaluated  within  2  weeks  Before  fit  After  taking  ASSET  course 
with  performance  tool  performing  a  2  incision  4  compartment 
fasciotomy  on  a  cadaver. 

-  ANOVA  (repeated  measures  for  pre  and  post-training) 

•  Pearson  correlations  for  time,  number  of  compartments 
decompressed,  anatomical  knowledge,  surface  landmarks,  and  IPS. 

•  Plotted  with  evaluation  data  from  8  expert  practicingtrauma 


Table  1:  Descriptive  statistics  for  Pre  and  Pos 


onent  and  IPS  scores  for  surgical  residents 


(F:  1.7)  (p<0.17) 


|f:  21.2)  (pOOOOOl) 


(F:  20.81  (pcO.OOOOl) 


(F:  18.9)  (p«0.00001) 


(F:  188)  lixOOOOOl) 


(F:  24.61  lixOOOOOl) 


Conclusions 

It  is  apparent  that  the  correct  identification  of  surface  landmarks  and  incisions  was  associated  more  successfully  decompressed  compartments.  By  this  measure,  structural 
recognition  during  specific  procedural  steps  and  surface  anatomic  knowledge  were  highly  impactful  information  even  when  taught  during  a  1-day  course.  This  training 
approach,  normally  acquired  during  the  entirety  of  residency  training,  may  help  accelerate  acquisition  of  emergency  surgery  specific  skills  to  compensate  for  shortened 
training  hours  or  when  just-in-time  training  is  necessary. 

While  there  is  no  doubt  that  this  repeated  deliberate  practice  of  unfamiliar  procedures  will  improve  skill  and  knowledge,  the  surgeons  were  also  given  no  guidance  or 
prompting  while  they  were  performing  the  procedures.  Performing  non-routine  procedures  completely  independently  during  residency  is  a  highly  unusual.  Although 
many  mistakes  were  made,  the  residents  also  felt  that  they  improved  from  learning  what  they  did  not  know  and  having  to  think  through  a  case  and  surgery  critically. 
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Appendix  16a:  In  Progress  Review  (August  2014) 


DEFENSE  HEALTH  AGENCY  -  JOINT  PROGRAM  COMMITTEE  - 1  (JPC-1) 

In  Progress  Review  (IPR)  August  2014  (Orlando,  FL) 

"USE  OF  PERFORMANCE  MEASURES  TO  EVALUATE,  DOCUMENT  COMPETENCE  AND  DETERIORATION  OF  ASSET  SURGICAL  SKILLS  -  MPI-PSD"  -  Univ.  of  Maryland; 
PI:  Dr.  Colin  Mackenzie 


1)  Strength(s)  of  Scientific  Methodology  / 
Approach 

2)  Strength(s)  of 
Investigative  Team's 
Risk  Identification  / 
Risk  Management 
Approach 

3)  Strength(s)  of 
Investigative  Team’s 
Schedule  / 
Performance  -  Goals, 
tasks,  milestones, 
and  deliverables 

4)  Strength(s)  of  Results  / 
Deliverables  /  Outcomes 
Regarding  respective  (a) 
needs  of  Military  /  End 
User  and  (b)  project's 
goals,  tasks,  milestones, 
and  deliverables 

5)  Recommendation,  based  on  how  well 
it  addresses  needs  of  the  DOD  or  any 
individual  agency  (e.g..  Army,  Air  Force, 
Navy,  Marines)  based  on  capability 
being  developed.  Identify  any  Military 
Organization,  Individual  Subject  Matter 
Expert,  End-User,  &/or  Advanced 
Developer  who  may  be  able  to 
accelerate  transition. 

6)  Overall  Global 

Score 

Score:  3  -  Good 

Score:  3  -  Good 

Score:  3  -  Good 

Score:  3  -  Good 

Met  essential  needs 

Score:  3 -Good 

Hypothesis:  1.  Metrics  can  be  used  to 
document  surgical  competence.  2.  The 
metrics  can  document  decay  of  skills  after 
five  years. 

RISK:  Q1.  Getting 
enough  ASSET  alumni 
to  take  the  ASSET 
course.  1 .  Develop 
performance  metrics  for 
surgeons  and  their 
skils  degradation  using 
the  ASSET  coiase. 

DONE  2.  Four  injuries: 
femoral  artery  injury, 
auxiiary  artery, 
compartment 
syndrome,  and 
fasci  atomy. 

1.  Develop 

performance  metrics  for 
surgeons  and  their 
skils  degradation  using 
the  ASSET  course 

DONE  2.  Four  injuries: 
femoral  artery  injury, 
auxiiary  artery, 
compartment 
syndrome,  and 
fasciotomy. 

Doing  before  and  after 
checklist  on  video  recordings 
on  residents  and  staff  doing 
the  ASSET  course 

Developed  a  new  metric 
(RTI)  to  score  the 
procedures. 

Score:  4  -  Outstanding 

Score:  3  -  Good 

Score:  3  -  Good 

Score:  4  -  Outstanding 

Met  essential  needs 

Score:  3  -  Good 

Well  stated  problem  to  be  addressed  skill 
decay  of  trauma  surgical  skils  vascular 
injuries.  Sound  methodological  approach. 

cohort  of  500  surgeons 
data  collection  after 
period  of  non-use 

seems  to  be  on 
schedule 

excefent  study  has  the 
potential  to  contribute 
significantly  to  patient  safety 

Significant  study  results  wiR  positive  impact 
combat  surgery. 

Excellent  study 
recommend  continue 
funding. 

Score:  3  -  Good 

Score:  3  -  Good 

Score.  3  -  Good 

Score:  3 -Good 

Met  essential  needs 

Score:  3  -  Good 

Seems  Use  JPC/DoO  is 
creating  several  automated 
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Appendix  17:  The  Assets  of  ASSET:  Improving  Surgical  Performance  Confidence  through 
an  Anatomy  Skills  Review  Course  for  Surgeons 


Evan  M  Garofalo1,  Stacy  Shackelford1,2,  Megan  A  Holmes1'3,  Colin  Mackenzie1, 
Mark  W  Bowyer4.  University  of  Maryland,  Baltimore,  MD,  2C-STARS,  Baltimore, 
MD,  3 Johns  Hopkins  University,  4USUHS,  Bethesda,  MD 

Rapid  control  of  major  hemorrhage  is  a  primary  goal  in  trauma  surgery.  However,  many 
surgeons  have  little  practical  experience  with  the  required  vascular  exposures.  To  address 
this,  the  American  College  of  Surgeons  developed  the  Advanced  Surgical  Skills  for 
Exposure  in  Trauma  (ASSET)  course  to  review  anatomy,  skills  and  techniques  for  major 
vascular  exposures.  Since  2008,  a  broad  range  of  participants  have  attended,  including 
surgeons  of  many  specialties,  deploying  military  surgeons  and  surgery  residents. 

We  compared  self-reported  confidence  of  participants  (n=562)  in  surgical  tasks  (n=47)  at 
baseline  and  directly  after  ASSET  training  to  examine  the  effect  of  the  course  stratified 
by  surgical  experience  level  (resident/fellow;  <8  years  post-residency;  8+  years  post¬ 
residency),  specialty  (trauma/vascular;  general  surgery;  other  specialties),  and  body 
region. 

Results  of  Freeman-Halton  3x2  tests  indicated  significant  gains  in  confidence  scores  for 
all  specialties  (p< 0.02),  particularly  for  general  surgeons  (p<0.01)  and  exposures  in  the 
chest  (p<0.001),  after  ASSET.  There  was  no  difference  in  confidence  gained  by  surgical 
experience.  This  study  demonstrates  the  value  of  continuing  education  in  applied 
anatomy  for  clinical  practice.  Given  the  frequency  of  vascular  trauma  in  current  military 
conflicts,  the  impact  of  ASSET  is  particularly  relevant  for  preparing  deploying  surgeons 
for  the  theatre. 


Funding  support  from:  US  Army  W81XWH- 13-2-0028 


The  assets  of  ASSET:  improving  surgical  performance  confidence  through  an 
anatomy  and  skills  review  course  for  surgeons. 
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Abstract  536.11/B45 

Rapid  control  of  major  hemorrhage  is  a  primary  goal  m  trauma  surgery  However,  many 
surgeons  have  little  practical  experience  with  the  required  vascular  exposures.  To  address 
this,  the  American  College  of  Surgeons  developed  the  Advanced  Surgical  Skills  for  Exposure 
In  Trauma  (ASSET)  course  to  review  anatomy,  skills  and  techniques  for  major  vascular 
exposures  Since  2010.  a  broad  range  of  participants  have  attended,  including  surgeons  of 
many  specialties,  deploying  military  surgeons  and  surgery  residents 

We  compared  self-reported  confidence  of  participants  (n=562)  m  surgical  tasks  (n=47)  at 
baselme  and  directly  after  ASSET  training  to  examine  the  effect  of  the  course  stratified  by 
surgical  experience  level  (resrfenlfellow  <8  years  post-residency,  8*  years  post-residency). 
I  trauma, 'vascular  general  surgery;  other  specialties),  and  body  region 


Results  of  Freeman-Halton  3x2  tests  indicated  significant  gams  in  confidence  scores  for  all 
specialties  [p< 0.02),  particularly  for  general  surgeons  (jxO.OI)  and  exposures  in  the  chest 
(p<0.001).  after  ASSET.  There  was  no  difference  in  confidence  gamed  by  surgical 
experience  This  study  demonstrates  the  value  of  continuing  education  in  applied  anatomy 
for  clinical  practice  Given  the  frequency  of  vascular  trauma  m  current  military  conflicts,  the 
impact  of  ASSET  is  particularly  relevant  for  prepanng  deploying  surgeons  for  the  theatre. 


Introduction 

Work  hour  restrictions  for  residents,  implemented  in  2003.  reduced  in-hospital  work  hours  to 
80  hoursfweek.'  Surgical  residents  are  performing  fewer  vascular  exposures  during  their 
surgrcaltraining 

•  .  average  cases  /  graduating  general  surgery  chief  residents  from  1999  to  2012*: 

Total  cases  60  4  to  35.5 
Major  vascular  procedures:  8  to  0.7 

The  Advanced  Surgical  Skills  for  Exposure  in  Trauma  (ASSET)  course  was  developed  to 
address  declining  residency  training  experience 

•  Launched  in  20 1 0  and  approved  by  Amencan  College  of  Surgeons 

•  1  day  cadaver-based  course  reviews  anatomy  and  surgical  skills  to  perform  ail  major 
vascular  exposures,  emphasizing  swift  exposure  as  vital  for  control  of  major  hemorrhage 

•  Part  of  military  pre-deployment  training  and  surgical  residency  programs  to  review 
trauma  surgical  skills. 

Research  question:  is  there  an  effect  of  the  level  of  surgical  experience  or  surgical 
specialty  on  confidence  to  perform  procedures  independently  after  ASSET  training? 

We  hypothesize  that  general  surgeons  (B  Specially)  and  Resident  /  Fellows  will  report  the 
greatest  improvement  in  confidence. 


Materials 

Enrollment  questionnaires  from  53  ASSET  courses  offered  nationally  from  2010  to  201 33 
to  gather  basic  professional  demographics  and  surgical  confidence  from  a  broad  range  of 
surgeons  (n=562)  Analysis  was  conducted  by  experience  level  and  surgical  specialty 

(Table  1). 

Surgical  confidence,  Pre  &  Post-ASSET:  ^ 

Self-reported  confidence  level  for  47  If  an 
procedures'  partitioned  into  upper  limb.  $  Jt  2! 
lower  limb,  chest,  neck,  pelvis  &  abdomen 
procedures  (5  point  liken  Scale’  tmo  || 
confidence  5-<n*»e  a  bl  of  confidence) 

Methods 


i  of  1-3:  low  confidence  and;  and  4-5:  high  confidence. 

Delta  for  median  confidence  to  perform  procedures  independently  Pre-  lo  Post-ASSET 

training  compared  by 

Body  region:  Wilcoxon  matched  pairs  test 

Direction  of  change:  (increase,  decrease,  stayed  the  same)  by  Experience.  Specialty  and 
Body  region  (Freeman-Halton  contingency  tests) 


Results 

Figures  1  &  2  report  the  average  number  of  select  procedures 
reportedly  performed  Pre-  confidence  levels  generally  align 
with  number  of  procedures  reportedly  performed  by  speoally 
and  experience  level 

Tables  2  &  3  detail  the  median  Pre-  4  Post-ASSET  confidence 
scores  by  experience  and  specialty.  Confidence  increased 
i  for  all  Pre-Post  pairs  at  o=0  .05 


Of  low  Pre-ASSET  confidence. 

•  39%  high  Post  confidence  for  all  procedures 
-  49%  hgh  Post  confidence  for  upper  limb  procedures 

Experience  Significant  effect  of  experience  on  Post-ASSET 
confidence,  for  all  body  regions  but  lower  limb  (Table  5). 
Residents  report  low  to  high  confidence  shifts  most  particularly 
for  lower  limb  and  pelvis  Senior  Attendings  (>8  years  practice) 
report  low  confidence  Pre  4  Post  training. 

Specialty  Significant  effect  of  specialty  on  Post-ASSET 
confidence,  for  all  regions  but  lower  limb  (Table  6). 

Most  improved  confidence:  General  Surgery  (B) 

Least  improved  body  region:  chest,  abdomen  4  pelvis 


Figure  I  Avarsge  number  ol  pscsdures  perform!  by  specialty  group 
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Appendix  18:  Current  List  of  Works 


Abstracts  for  Presentation/Publication 

1 .  The  assets  of  ASSET :  Improving  surgical  performance  confidence  through 
anatomy  and  skills  review  course  for  surgeons 

(Poster:  Federation  of  American  Societies  for  Experimental  Biology  (April  2014) 
Authors:  Evan  Garofalo,  Stacy  Shackelford,  Megan  Holmes,  Colin  Mackenzie, 
Mark  Bowyer 

2.  Development  of  a  Trauma  Readiness  Metric  Score  for  Surgeons 

(Podium:  Military  Health  Services  Research  Symposium  (MHSRS,  August  2014) 
Authors:  Evan  Garofalo,  Stacy  Shackelford,  Valerie  Shalin,  Megan  Holmes, 

Jason  Pasley,  Elliot  Jessie,  Babak  Sarani,  Sharon  Henry,  Mark  Bowyer,  Colin 
Mackenzie 

3.  Mobile  Platform  to  Evaluate  Individual  Surgeon  Technical  Skills 

(Poster:  Military  Health  Services  Research  Symposium  MHSRS,  August  2014) 
Authors:  Colin  F  Mackenzie,  Mark  Fitzgerald,  Kon  Mouzakis,  Joost  Funke 
Kupper,  George  Hagegeorge,  Peter  Hu,  Evan  Garofalo,  Mark  Bowyer,  Sharon 
Henry,  Stacy  Shackelford 

4.  Mobile  Platform  to  Evaluate  Individual  Technical  Skills 

(Presentation:  American  Society  of  Anesthesiologists  (ASA,  October  2014)) 
Authors:  Colin  F  Mackenzie,  Mark  Fitzgerald,  Kon  Mouzakis,  Joost  Funke 
Kupper,  George  Hagegeorge,  Peter  Hu,  Evan  Garofalo,  Mary  Njoku,  Stacy 
Shackelford 

5.  Evaluation  of  individual  surgeon  technical  skills  during  four  emergency 
procedures  (Podium  Presentation:  Association  of  Military  Surgeons  of  US 
(AMSUS,  December  2014)) 

Authors:  Colin  F  Mackenzie,  Evan  Garofalo,  Hegang  Chen,  Valerie  Shalin,  Kristy 
Pugh,  Stacy  Shackelford,  Sharon  Henry,  Mark  Bowyer,  Mark  Fitzgerald,  Joost 
Funke  Kupper,  George  Hagegeorge,  Peter  Hu,  Kon  Mouzakis 

6.  Anatomy  in  the  performance  of  a  lower  extremity  fasciotomy  before  and 
after  training(Publication:  Journal  of  Surgical  Education  2014) 

Evan  Garofalo,  Stacy  Shackelford,  Valerie  Shalin,  KristyPugh,  Hegang  Chen, 
Jason  Pasley,  Babak  Sarani,  Sharon  Henry,  Mark  Bowyer,  Colin  Mackenzie 

7.  Accurate  assessment  of  surgical  skill  improvements  after  training 

(Presentation:  Eastern  Association  of  Surgery  for  Trauma  (EAST,  January  2015)) 
Authors:  Stacy  Shackelford,  Evan  Garofalo,  Valerie  Shalin,  Kristy  Pugh,  Jason 
Pasley,  Babak  Sarani,  Sharon  Henry,  Mark  Bowyer,  Colin  Mackenzie 


8.  Management  of  Vascular  trauma  by  senior  surgical  residents:  Perception 
does  not  equal  reality 

(Presentation:  Academic  Surgical  Congress  (ASC  February  2015)) 

Authors:  Mark  Bowyer,  Stacy  Shackelford,  Evan  Garofalo,  Kristy  Pugh,  Colin 
Mackenzie 

9.  Assessment  of  surgical  anatomy  skills  in  upper  and  lower  limb  vascular 
control  and  before  and  after  training 

(Podium:  Association  for  Surgical  Education  (April  2015)) 

Authors:  Evan  Garofalo,  Stacy  Shackelford,  Valerie  Shalin,  Kristy  Pugh,  Hegang 
Chen,  Jason  Pasley,  Babak  Sarani,  Sharon  Henry,  Mark  Bowyer,  Colin 
Mackenzie 

1 0.  Mobile  Platform  for  Assessing  Emergency  Trauma  Surgical  Skill 
Performance 

(Poster:  Association  for  Surgical  Education  (April  2015)) 

Authors:  Colin  Mackenzie,  Stacy  Shackelford,  Evan  Garofalo,  Hegang  Chen, 
Jason  Pasley,  Sharon  Henry,  George  Hagegeorge,  Kristy  Pugh,  Mark  Bowyer. 
STAR  and  Shock  Trauma  Center,  Departments  of  Anatomy  and  Epidemiology, 
University  of  Maryland  School  of  Medicine,  USAF  and  USUHS  Bethesda  MD 

1 1  .What’s  on  the  surface  matters  too:  assessment  of  surgical  anatomy  and 
successful  performance  of  a  lower  extremity  fasciotomy  before  and  after 
training 

(Poster:  Federated  American  Societies  for  Experimental  Biology,  American 
Association  of  Anatomists  (FASEB,  AAA  March  2015)) 

Authors:  Evan  Garofalo,  Stacy  Shackelford,  Valerie  Shalin,  Kristy  Pugh,  Hegang 
Chen,  Jason  Pasley,  Babak  Sarani,  Sharon  Henry,  Mark  Bowyer,  Colin 
Mackenzie 

1 2.  Anatomic  knowledge  increases  after  participation  in  ASSET  training 

(Poster:  American  Association  of  Clinical  Anatomists  2015) 

Authors:  Kristy  Pugh,  Evan  Garofalo,  Brandon  Bonds,  Colin  Mackenzie 

1 3.  Using  an  Individual  Procedure  Score  (IPS)  before  and  after  the  ASSET 
course  training  to  benchmark  a  hemorrhage-control  performance  metric 

(Publication:  Journal  of  Surgical  Education2015) 

Authors:  Colin  Mackenzie,  Evan  Garofalo,  Stacy  Shackelford,  Valerie  Shalin, 
Kristy  Pugh,  Hegang  Chen,  Adam  Puche,  Jason  Pasley,  Babak  Sarani,  Sharon 
Henry,  Mark  Bowyer 


14.  How  successful  is  ASSET  at  training  residents  in  lower  extremity  fasciotomy 
compared  to  trauma  surgeons? 

(Podium:  Military  Health  System  Research  Symposium  2015) 

Authors:  Evan  Garofalo, Mark  Bowyer,  Stacy  Shackelford,  Valerie  Shalin,  Kristy 
Pugh,  Hegang  Chen,  George  Hagegeorge,  Babak  Sarani,  Jason  Pasley,  Sharon 
Henry,  Colin  Mackenzie 

1 5.  Which  combat  surgical  skills  degrade  after  training? 

(Podium:  Military  Health  System  Research  Symposium  2015) 

Authors:  Stacy  Shackelford,  Evan  Garofalo,  Valerie  Shalin,  Kristy  Pugh,  Hegang 
Chen,  George  Hagegeorge,  Jason  Pasley,  Sharon  Henry,  Mark  Bowyer,  Colin 
Mackenzie 

1 6.  Are  physical  model  assessments  comparable  to  cadaver  for  combat  surgical 
technique? 

(Podium:  Military  Health  System  Research  Symposium  2015) 

Authors:  Brandon  Bonds,  Evan  Garofalo,  Kristy  Pugh,  Mark  Bowyer,  Stacey 
Shackelford,  Colin  Mackenzie 

Peer  Reviewed  Journal  Articles 

1.  Development  and  validation  of  trauma  surgical  skills  metrics:  Preliminary 
assessment  of  performance  after  training 

(Publication:  Journal  of  Trauma  and  Acute  Care  Surgery  (July  2015)) 

Authors:  Stacy  Shackelford,  Evan  Garofalo,  Valerie  Shalin,  Kristy  Pugh,  Hegang 
Chen,  Jason  Pasley,  Babak  Sarani,  Sharon  Henry,  Mark  Bowyer,  Colin 
Mackenzie 

2.  Perception  does  not  equal  reality  for  resident  vascular  trauma  skills 

(Publication:  Journal  of  Surgical  Research  (October  2015)) 

Authors:  Mark  Bowyer,  Stacy  Shackelford,  Evan  Garofalo,  Kristy  Pugh,  Colin 
Mackenzie 

3.  Using  an  individual  procedure  score  before  and  after  the  Advanced  Surgical 
Skills  Exposure  for  Trauma  Course  training  to  benchmark  a  hemorrhage- 
control  performance  metric 

(Publication:  Journal  of  Surgical  Education  (November/December  2015)) 
Authors:  Colin  Mackenzie, Evan  Garofalo,  Stacy  Shackelford,  Valerie  Shalin, 
Kristy  Pugh,  Hegang  Chen,  Adam  Puche,  Jason  Pasley,  Babak  Sarani,  Sharon 
Henry,  Mark  Bowyer 

4.  Computer-assisted  video  hand-motion  analysis  combined  with  image 
identification  as  a  step  to  automated  evaluation  of  surgeon  technical 
performance:  a  preliminary  report  of  methodology  and  analyses 


(Publication:  (in  review)  Ergonomics) 

Authors:  Colin  Mackenzie,  Darcy  Watts,  Rajan  Patel,  Shiming  Yang,  George 
Hagegeorge,  Evan  Garofalo,  Peter  Hu,  Adam  Puche,  Valerie  Shalin,  Kristy 
Pugh,Guin  Granite,  Lynn  Stansbury,  Stacy  Shackelford,  Sam  Tisherman 

5.  Trauma  training  courses  for  surgeons  and  validation  of  their  efficacy 

(Publication:  (in  review)  J.  Am  Coll  Surg) 

Authors:  Colin  Mackenzie,  Kristy  Pugh,  SamTisherman,  Stacy  Shackelford,  Mark 
Bowyer. 

6.  Sensor-free  Computer-Vision  hand-motion  entropy  and  video-analysis  of  technical 
performance  during  open  surgery:  proof  of  concept  report  of  methodology  and 

analysis 

Accepted  for  Publication  Human  Factors  and  Ergonomics 

Colin  F  Mackenzie,  Darcy  Watts,  Rajan  Patel,  Shiming  Yang,  George  Hagegeorge, 
Evan  Garofalo,  Peter  F  Hu,  Adam  Puche;  Kristy  Pugh,  Guinevere  Granite,  Lynn  G 
Stansbury,  Stacy  Shackelford,’  Samuel  Tisherman,  Valerie  Shalin 
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Appendix  19:  Full  Proposal  Invitation  Letter  2015  for  BA150077  -  “Refreshing  Combat  Surgical 
Skills  for  Vascular  Control” 


DEPARTMENT  OP  THE  ARMY 
US  ARMY  MEDICAL  RESEARCH  ACQUISITION  ACTIVITY 
S3G  CHANDLER  STREET 
FORT  DfTmCKMD  il7U!-5Ul4 


January  15.  2015 

SUBJECT:  BA15D077  -  ''Refreshing  Combat  Surgical  Skills  for  Vascular  Control’ 


Colin  Mackenzie 
University  of  Maryland,  Baltimore 
419  Redwood  Street,  Suite  225 
Baltimore,  MD  21201 


Dear  Dr.  Mackenzie: 

You  are  invited  to  submit  a  propgsabapplicationto  the  Fiscal  Year  2015  (FY1 5) 
Department  of  Defense  (DoD)  Broad  Agency  Announcement  (BAA)  for  Extramural 
Medical  Research.  The  reviewers  found  your  pre-proposalfp  re-application  to  be  of 
interest  to  US  Army  Medical  Research  and  Materiel  Command  { JSAMRMC)  programs 
aimed  at  the  solution  of  medical  problems  of  military  importance. 

In  accordance  with  the  Federal  Acquisition  Regulations,  the  USAMRMC  advertised 
for  research  and  development  proposals/appiications  using  a  BAA.  Since  your 
proposal/applicatiun  will  be  submitted  after  October  1.  2014.  you  are  required  to  submit 
it  under  t fie  FY15  BAA  (released  on  October  1,  2014at  www.grants.gov  under  CFDA 
12.420)  and  to  use  the  forms  identified  in  the  announcement.  Attempts  to  submit  any 
forms  from  a  previous  iteration  of  tha  DoD  BAA  wilt  be  rejected  by  Grants.gov. 

in  addition,  your  proposal/applicatlon  must  Include  the  requisite  components,  comply 
with  preparation  instructions,  include  the  log  number  BA150Q77,  and  be  submitted  by 
11:59  p.m.  Eastern  time  on  April  15,  2015  as  described  in  the  FY15  BAA  and  General- 
Sub  mission  Instructions. 

Please  note  that  your  proposaltepplication  may  be  rejected  for  administrative 
reasons  without  further  review  if  the  budget  or  research  project  as  described  in  the 
pre-proposal/pre-application  differs  significantly  from  the  full  propose Vappli cation . 

Based  on  the  information  provided  in  the  pre-application,  you  should  address  the 
fallowing  in  your  application:  the  inclusion  of  a  trauma  and  combat  surgeon  or 
equivalent  expert  lo  advise  on  this  study,  detailed  statistical  design  including  power 
analysis  accounting  for  recuifment  and  retention  of  subjects,  and  justification  for  the 
proposed  period  of  performance. 
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Appendix  20:  Dr.  Mackenzie  Doctoral  Candidacy  Presentation 


Title:  Assessment  of  Surgical 
Performance: 

Early  Stage  Assessment 

Colin  F  Mackenzie  MB  ChB 

Shock  Trauma  Anesthesiology 
Research  Center  (STAR), 

University  of  Maryland  School  of 
Medicine 

cmack003@unnaryland.edu 
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Appendix  21:  Request  for  Budget  Modification  with  Justification  March  2015 


March  31“  2015 


Dear  Ms.  Bane, 

I  would  like  to  request  approval  of  some  budget  modifications  to  W81XWH-13-2-008 

“Use  of  Performance  Measures  to  Evaluate,  Document  Competence  and  Deterioration  of 
Advanced  Surgical  Skills  Exposure  for  Trauma  (ASSET)  Surgical  Skills"  The  Title  was 
abbreviated  as  Retention  and  Assessment  of  Surgical  Performance  (RASP) 

PRINCIPAL  INVESTIGATOR:  Dr  Colrn  Mackenzie 

These  budget  modifications  will  not  change  the  overall  cost  of  the  project,  nor  will  they  change 
the  scope  of  work  or  the  deliverables.  This  modification  is  intended  to  optimize  the  allocation  of 
the  remaining  available  funds  for  this  project  to  ensure  the  best  deliverable  products  of  this 
overall  effort. 

In  general  therefore,  this  budget  modification  requests  re-allocation  approval  for  salary  support 
of  additional  experts  to  assist  with  data  analysis  and  interpretation  and  to  account  for  salary 
increases  of  the  existing  investigators,  and  additional  equipment  to  use  newly  identified  tools  to 
interpret  these  data 

The  justification  for  the  requested  budget  changes  is  shown  below  and  the  source  of  funds  is 
$737,712.60  funds  as  outlined  m  the  attached  budget  showing  the  funds  remaining 

Carry-over  of  funds  occurred  in  Year  2  due  to  factors  below  1)  Travel  re-imbursement  requests 
for  Phase  1  and  Phase  2  candidates  were  less  than  budgeted.  2)  More  military  surgeons  than 
expected  were  enrolled,  so  stud}'  participation  payments  were  less  as  no  payments  made  to  these 
military  surgeon  enroll ees  Military  surgeons  cannot  accept  stud}'  participation  payments  for  US 
Army  funded  studies.  3)  We  did  not  get  an  earlier  salary  increase  approved  for  investigators 
because  of  a  freeze  on  salaries  in  the  University  of  Maryland.  Some  of  the  cany  over  funds  will 
be  used  to  cover  salary'  increases.  4)  The  PI  has  obtained  additional  funding  (starting  July  2015) 
and  will  have  to  reduce  funded  effort  on  the  project  between  July  2015  until  project  end  March  1 
2016. 

We  request  modification  of  the  budget  to  use  these  funds  as  follows: 

1)  Increase  m  expenditure  on  Salary:  Salary'  costs  will  increase  beyond  budget  because  salary 
increases  have  been  approved  for  Drs  Garofalo  and  Mackenzie  and  Ms  Kristy'  Pugh , 
effective  July  1“  2015  Dr  Garofalo  is  funded  at  100%  effort  as  the  Research  Coordinator  of 
this  project  effort  and  is  an  Assistant  Professor  in  the  Department  of  Anatomy  at  the 
University  of  Maryland  School  of  Medicine.  As  of  July  l“  2015  we  would  like  approval  to 
increase  Dr  Garofalo’s  salary'  by  14.9  %  to  $79,374  for  her  excellent  performance  and  high 
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Appendix  22:  Schedule  for  Advisory  Board  Meeting  2015 


RASP1  ASSET  Advisory  Board  Meeting  2015 

Attendees:  Advisory  Board  Members;  James  Shanteau  PhD.  Nick  Sevdaiis  PhD,  Valerie 
Shalin  PhD,  Guests:  Kevin  Kunkler  MD.  William  Chin  MD  FACS,.  Project  Investigators  : 

Colin  Mackenzie  MD  (Project  PD  ,  Col  (Rtd)  Mark  Bowyer  MD  FACS  (USUHS) ,  Col 
Stacy  Shackelford  MD  FACS  (USAF ISR),  Maj  Jason  Pasely  DO,  FACS  (USAF  C-STARS 
Baltimore),  Shiming  Van  PhD.  Research  Staff:  Guin  Granite  PhD.  Kristy  Pugh  MS, 
Support  Staff;  Barbara  McGee.  Telecof  Attendee:  COR  Tony  Story  USAMRMC 


Advisory  Board  Meeting  at  UNIVERSITY  OF  M.ARYLAND,  October 

2015 

RETENTION  &  ACQUISITION  OF  SURGEON  PERFORMANCE  (R.4SP) 

Date;  Thursday  October  J*:  Time:  8  am-  5 pm 

Place:  Room  TIR12  Shock  Trauma  Directors  Conference  Room,  Baltimore  \£D  21201(410-627-5616) 

Agenda  Day  1  of  RASP  Advisory  Board  meeting:- 

8am-  8:30  am  Coffee  and  Breakfast  Pastries 

8:30  am  -  8  :45  am:  Welcome  and  Self-  Introductions 

8:45  am  —  9:15  am  Overview  of  Surgeon  Performance  study  Objectives  Hypotheses  and  Data 
Collection:  Colin  Mackenzie 

9  :15  am  -  9:45  am  Results:  Performance  effect  of  ASSET  training:  Colin  Mackenzie 
9:45  am  -  10  am  REFRESHMENT  BREAK 

10  am  -  11  am  Results:  Skill  degradation  compared  to  immediate  post-ASSET  training  and  expert 
performance.  Stacy  Shackelford  and  Colin  Mackenzie 

11am-  11:45  am:  Ray  Perez:  “ONR  interests  in  performance  and  learning" 

11:45  am  -  11  noon:  Discussion 

12  Noon:  LUNCH  BREAK.  Tour  Shock  Trauma  Resuscitation  Unit  and  Simulation  Center  (  if 
interested). 

lpm-  1:30pm  Kevin  Kunkler:  US  Army  JPC'-l:  Technical  Skills  retention  and  Refreshing 

1:  30  pm:  Valerie  Shalin,  Eric  Robinson:  "Planning  vs.  execution:  detecting  qualitative  differences 
in  surgical  skills" 
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Appendix  23:  Full  Proposal  Invitation  Letter  2016  for  BA15808  -  “Emergency  Refreshing  of 
Combat  Surgical  Skills” 


DEPARTMENT  OF  THE  ARMY 

U  S  ARMY  MEDICAL  RESEARCH  ACQUISITION  ACTIVITY 
820  CHANDLER  STREET 
FORT  DETRICK  MD  21702-5014 


December  21,  2015 


SUBJECT:  BA1 50808  -  "Emergency  Refreshing  of  Combat  Surgical  Skills" 


Colin  Mackenzie 

University  of  Maryland  Baltimore 

IIS.  Paca  Street 

Lower  Level 

Baltimore,  MD  21201 


Dear  Dr.  Mackenzie: 

You  are  invited  to  submit  a  proposal/application  to  the  Fiscal  Year  2016  (FY16) 
Department  of  Defense  (DoD)  Broad  Agency  Announcement  (BAA)  for  Extramural 
Medical  Research.  Your  pre-proposal/pre-application  was  reviewed  by  the  U  S.  Army 
Medical  Research  and  Materiel  Command  (USAMRMC)  scientific  staff  Invitations  to 
submit  full  proposals/applications  were  based  upon  technical  merit,  programmatic 
considerations,  and  availability  of  funds.  The  reviewers  found  your  pre-proposal/pre- 
application  to  be  of  interest  to  the  USAMRMC  programs  aimed  at  the  solution  of 
medical  problems  of  military  importance. 

In  accordance  with  the  Federal  Acquisition  Regulations,  the  USAMRMC  advertised 
for  research  and  development  proposals/applications  using  a  BAA. 

You  are  required  to  submit  a  proposal/application  under  the  FY16  BAA  (released  on 
October  1,  2015  on  Grants.gov  fhttD://www  arants.oov/1  under  CFDA  12.420)  and  to  use 
the  forms  identified  in  the  announcement.  Attempts  to  submit  any  forms  from  a 
previous  iteration  of  the  DoD  BAA  will  be  rejected  by  Grants.gov. 

Your  proposal/application  must  include  the  requisite  components,  comply  with 
preparation  instructions,  include  the  log  number  BA1 50808,  and  be  submitted  by  11:59 
p.m.  Eastern  time  on  March  21 , 2016  as  described  in  the  FY16  BAA  and  General 
Submission  Instructions. 

Please  note  that  your  proposal/application  may  be  rejected  for  administrative 
reasons  without  further  review  if  the  budget  or  research  project  as  described  in  the  pre- 
proposal/pre-application  differs  significantly  from  the  full  proposal/application. 

Proposals/Applications  must  be  submitted  by  the  Authorized  Organizational 
Representative  through  Grants.gov  (www  qrants.gov)  For  synopsis  details,  full  BAA, 
and  proposal/application  package  including  instructions,  go  to 
httDV/www  arants  aov/web/arants/search-grants  htrnl  and  enter  Funding  Opportunity 
Number  W81XWH-16-R-BAA1  under ‘Basic  Search  Criteria." 
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Appendix  24:  International  Meeting  for  Simulation  in  Healthcare  (IMSH)  2016  Agenda,  Summary, 
&  Presentation 


1300-1305 

International  Meeting  for  Simulation  in  Healthcare  (IMSH)  2016 

January  18,  2016 

Department  of  Defense  Special  Session  -  Program  Schedule 

SSiH  Welcome  and  Introduction  (Davna  Downing:  Gerald  R  Moses) 

1305-1330 

Current  Projects  &  Future  Vision  (K  K tinkler) 

1330-1400 

Research  Presentations  (15  mm'presentation): 

MODERATOR:  K  Kunkler 

1.  Development  of  Direct  Observation  and  Automated  Assessment  Tools  for  Multiple-Casualty 

Scenarios  (J.  Lopreiato) 

2.  Autonomous  Mentoring  Systems  far  Procedural  Shill  Acquisition  and  Assessment  (G.  Miller) 

1400-1415 

BREAK 

1415-1515 

Skills  Decay  Research  Presentations  (15  mi  representation): 

MODERATORS:  K  KUNKLER  R  PEREZ 

1 .  Skill  Degradation  Evaluation  Toolkit  far  Eliminating  Competency-loss  Trends  (SkiQ- 
DETECT)  (E.  Baker) 

2.  Psycho-Motor  and  Error  Enabled  Simulations:  Modeling  Vulnerable  Skills  m  the  Pre- 
Masterv  Phase  (C.  Pugh) 

3.  Use  of  Performance  Measures  to  Evaluate.  Document  C ompetence  and  Deterioration  of 

ASSET  Surgical  Skills  (C.  MacKenzie) 

4.  Surgical  Skills  Training  and  Assessment  Instrument  (SUSTAIN)  (A  Skinner) 

1515-1545 

Expert  Panel:  Understanding  and  Minimizing  Skill  Decay'1  (Dr.  Pb  WF.Z  LEADS) 

1545-1600 

BREAK 

1600-1715 

Research  Presentations  (15  mm  presentation) : 

MODERATOR  H.  MAGEE 

1.  Results  of  Development  and  Testing  of  a  Conversational  Virtual  Patient  for  Healthcare  (T. 

Talbot) 

2.  BioGears:  Designing  and  Building  an  Extensible,  Modular,  Open  Source  Human  Physiology 

Engine  (J.  Webb) 

3.  Human  Tissue  Characterization  C  omparing  Properties  of  Human  vs  Simulated  Tissues  (J. 

Norfleet) 

4.  STOMP  -  Simulation  Training  for  Operational  Medicine  Providers  (M.  Spooner) 

5.  The  Development  of  the  Objective  Assessment  of  the  Effectiveness  of  Military  Medical 

Training  (A.  LaPotta) 

1715-1730 

Questions/Discussum  Adjourn 
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Acquisition  &  Retention  of  Trauma  Surgical 

Skills 

Colin  Mackenzie  &  RASP  Group*+ 

Shock  Trauma  Anesthesiology  Research  Center,  University  of 
Maryland  School  of  Medicine ,  Baltimore 

RASP  Group:  Stacy  Shackelford,  Sam  Tisherman,  Adam  Puche , 

Valerie  Shalin,  Hegang  Chen,  Kristy  Pugh,  Evan  Garofalo,  Guinevere 
Granite,  George  Hagegeorge,  Sharon  Henry,  Mark  Bowyer, 

Nyaradzo  Longinaker,  Peter  Hu,  Shiming  Yang,  Mayur  Narayan, 

Elliot  Jessie,  Jason  Pasley,  Babak  Sarani,  Niki  Squires,  Amechi 
Anazodo,  Brandon  Bonds,  William  Teeter,  Ann  Ro  mag  noli ,  Eric 
Robinson,  Megan  Holmes,  Alexys  Monoson,  Joseph  Pielago 

+  Funding:  U.S.  Army  Medical  Research  &  Materiel  Command 
W81XWH-13-2-0028. 


Appendix  25:  Anatomic  Knowledge  Increases  after  Participation  in  ASSET  Training 


PUGH,  Kristy  R.,  Evan  M.  GAROFALO,  Brandon  W.  BONDS,  and  Colin  F.  MACKENZIE. 
Shock  Trauma  &  Anesthesiology  Research  Center  &  Department  of  Anatomy  and  Neurobiology. 
University  of  Maryland  School  of  Medicine,  Baltimore,  MD  21201,  USA. 

INTRODUCTION.  Rapid  open  vascular  exposure  and  repair  can  be  life-saving  in  both  civilian 
and  combat  casualty  situations  and  requires  mastery  of  anatomic  knowledge,  which  is  generally 
built  by  repetition  in  training.  With  limitations  in  training  hours,  the  opportunities  for  surgeons  to 
gain  proficiency  in  performing  open  vascular  exposures  have  been  reduced.  The  Advanced 
Surgical  Skills  for  Exposure  in  Trauma  (ASSET)  course  is  a  training  program  that  teaches 
surgeons  over  forty  key  vascular  exposures  using  cadaveric  models.  We  hypothesized  that 
ASSET  training  improves  anatomic  knowledge.  METHODS.  Surgical  residents  were  evaluated 
by  two  co-located  evaluators  on  four  selected  ASSET  procedures  (axillary,  brachial  and  femoral 
artery  exposure,  and  lower  extremity  fasciotomy)  using  previously  validated  metrics  including 
anatomic  knowledge,  landmarks,  and  skin  incision  location  before  and  after  ASSET  training.  Pre 
and  post-ASSET  scores  were  compared  using  paired  t-test,  p<0.05  was  significant.  SUMMARY. 
Forty  surgical  residents  with  an  average  of  3.6  clinical  years  of  experience  were  evaluated.  Mean 
scores  for  anatomic  knowledge  were  significantly  improved  after  training  (Pre=  40.5  ±13.2  vs. 
Post=  69.8  ±  10.8;  p<0.00001.)  CONCLUSION.  Anatomic  knowledge  for  the  exposure  of 
traumatic  vascular  injuries,  where  mistakes  and  delays  can  have  dire  consequences,  benefits 
significantly  from  ASSET  training.  Supplemental  courses,  such  as  ASSET,  provide  alternative 
surgical  training  to  allow  infrequently  performed  procedures  to  be  practiced. 

(Sponsored  by  Grant  N0.W8IXWH- 13-2-0028  from  USAMRMC  JCP-1  and  CDMRP) 
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Appendix  26:  Using  an  Individual  Procedure  Score  (IPS)  Before  and  After  the  ASSET  Course 
Training  to  Benchmark  a  Hemorrhage-Control  Performance  Metric 


ORIGINAL  REPORTS 

Using  an  Individual  Procedure  Score  Before 
and  After  the  Advanced  Surgical  Skills 
Exposure  for  Trauma  Course  Training  to 
Benchmark  a  Hemorrhage-Control 
Performance  Metric  *  "r 


Colin  F.  Mackenzie,  MBChB,  *'r  Evan  Garofalo,  PhD ,*'*  Stacy  Shackelford,  MD,  FACS* 

Valerie  Shalin,  PhD,*  Kristy  Pugh,  MS,  *  Hegang  Chen,  PhD,*  Adam  Puche,  PhD* 

Jason  Pasley,  DO,  FACS,  Babak  Sarani,  MD,  FACS,  Sharon  Henry,  MD,  FACS,** 
and  Mark  Bowyer,  MD,  FACSf1 

'Shock,  Trauma  and  Anesthesiology’  Research  Center,  University  at  Maryland  School  ol  Medicine,  Baltimore, 
Maryland;  ’Department  af  Anesthesiology,  University  of  Maryland  School  of  Medicine,  Baltimore,  Maryland; 
*Depar%nent  af  Anatomy  and  Neuro biology,  University  ol  Maryland  School  ol  Medicine,  Baltimore, 
Maryland;  *US  Airforce  Center  for  he  Sustainment  of  Troumo  and  Readiness  Skils,  Baltimore,  Maryland; 
"Department  of  Psychology,  Wright  Slate  University,  Dayton,  Chta;  'Department  of  Epidemiology,  University 
of  Maryland  School  of  Medicine,  Baltimore,  Maryland;  Department  af  Sirgery,  George  Washington 
University  School  of  Medicine,  Washington,  District  of  Columbia;  "Department  af  Surgery,  University  af 
Maryland  School  af  Medicine,  Baltimore,  Maryland;  and  "Department  of  Surgery,  Unformed  Services 
University  ol  the  Health  Sciences,  Belhesda,  Maryland 


OBJECTIVE:  Tea  with  an  individual  procedure  score  (IPS) 
to  assess  whether  an  un preserved  cadaver  trauma  training 
course,  including  upper  and  lower  limb  vascular  exposure, 
imp  rows  correct  identification  of  surgical  landmarks,  under¬ 
lying  anatomy,  and  shortens  time  to  vascular  control. 

DESIGN:  Protective  study  performance  of  3  vascular 
exposure  and  control  procedures  (axilary.  brachial,  and  femcsal 
armies)  using  IPS  metrics  by  2  colocated  and  trained  evduatons 
before  and  after  training  with  the  Advanced  Surgical  SkJk 
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Exposure  for  Trauma  (ASSET)  course.  IPS,  including  idenb- 
beat  an  of  anattxmaJ  landmarks,  mcoatxis,  underlying  struc¬ 
tures.  and  time  t>  correction  of  each  procedure  was  compared 
before  and  alter  training  using  repeated  meaunsnent  madek. 

SETTING:  Audio-video  instrumented  cadaver  laboratory  at 
University  of  Maryland  School  of  Medicine. 

PARTICIPANTS:  A  total  of  41  second  to  sixth  year  surgical 
residents  from  surgical  programs  throughout  Mid-Adandc 
States  who  had  not  previously  taken  the  ASSET  course  were 
enrolled.  40  completed  the  pre-  and  post -ASSET  perform¬ 
ance  evaluations. 

RESULTS:  After  ASSET  training;  all  components  of  IPS 
increased  and  time  shortened  for  each  of  the  3  artery 
exposures.  Procedure  steps  performed  correctly  increased 
57%.  anatomical  knowledge  increased  43%  and  skin 
incision  to  passage  of  a  vessel  loop  twice  around  the  correct 
vessd  decreased  by  a  mean  of  2.5  minutes.  An  oxer  all 
vascular  trauma  readmes  index,  a  comprehensive  IPS  score 
for  3  procedures  increased  28%  with  ASSET  Training. 

CONCLUSIONS:  Improved  knowledge  of  surface  land¬ 
marks  and  underlying  anatomy  is  associated  with  increased 
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Appendix  27:  How  Successful  is  ASSET  at  Training  Residents  in  Lower  Extremity  Fasciotomy 
Compared  to  Trauma  Surgeons 


Evan  M  Garofalo,  PhD1,  Mark  Bowyer,  MD2,  Stacy  Shackelford,  MD3,  Valerie  Shalin, 
PhD4,  Kristy  Pugh,  MS1,  Hegang  Chen,  PhD1,  George  Hagegeorge1,  Babak  Sarani,  MD5, 
Jason  Pasley,  DO3,  Sharon  Henry,  MD1,  Colin  Mackenzie,  MBChB1 

'Department  of  Anatomy  and  Neurobiology,  University  of  Maryland  School  of  Medicine 

2 

"The  Norman  M.  Rich  Department  of  Surgery,  Uniformed  Services  University  of  Health 
Sciences 

USAF  Center  for  Sustainment  of  Trauma  and  Readiness  Skills,  Baltimore,  Maryland 
4Department  of  Psychology,  Wright  State  University,  Ohio 

5Shock  Trauma  Anesthesiology  Research  Center,  University  of  Maryland  School  of  Medicine 
^’Department  of  Epidemiology,  University  of  Maryland  School  of  Medicine 
5Center  for  Trauma  and  Critical  Care,  George  Washington  University 

o 

Department  of  Surgery,  University  of  Maryland  School  of  Medicine 

Acquiring  and  maintaining  combat  casualty  specific  surgical  skills  is  challenging  for  military  and 
civilian  surgeons.  Due  to  multiple  factors,  the  cases  of  major  vascular  trauma  reported  by 
graduating  chief  surgeons  in  the  last  decade  have  reduced  by  approximately  half  and  the  average 
cases  reported  for  lower  extremity  fasciotomy  (FAS)  is  1.2.  However,  specific  skill  training 
programs  have  been  developed  to  supplement  gaps  in  training  opportunities.  One  such  course  is 
the  Advanced  Surgical  Skills  Exposure  for  Trauma  (ASSET)  course,  created  to  fill  this  training 
capability  gap  and  includes  a  fasciotomy  module.  Incomplete  or  delayed  decompression  has 
contributed  to  loss  of  life  and  limb  in  the  current  conflicts.  Here  we  hypothesize  that  significantly 
improved  post-training  skills  in  the  performance  of  lower  extremity  fasciotomy  will  show 
evidence  of  degradation  through  time  without  further  practice. 

Methods:  We  evaluated  the  performance  of  lower  extremity  FAS  with  the  same  objective 
procedural,  knowledge,  and  subjective  psychomotor  skills  metrics  for  a)  third  to  sixth  year 
surgical  residents  before  and  after  ASSET  training  (n=39);  b)  current  skill  in  surgeon  trained  2-5 
years  previously  (n=  11);  and  c)  expert  trauma  surgeons  (n=8).  The  objective  assessment  is  a 
case-based  scenario  with  checklists  for  technical  and  non-technical  skill  components  including: 
knowledge  (diagnostic/management,  anatomical  including  surface  landmarks,  incision 
placement,  and  structure  identification),  procedural  steps,  technical  skills  (subjective 
psychomotor  tissue  handling)  and  procedure  time.  Scores  were  calculated  for  each  component 
and  as  a  composite  Individual  Procedure  Score  (IPS).  Assessments  were  conducted  by  two  co¬ 
located  trained  evaluators  on  unpreserved  cadavers.  Performance  evaluations  were  compared 
before  and  after  ASSET  and  between  surgeon  groups  using  repeated  measures  ANOVA. 
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Results:  ASSET  training  significantly  improved  performance  in  surgical  residents  for  all 
measures  except  time  (all  ANOVAs  /;<(). 0006).  The  average  number  of  compartments 
successfully  decompressed  increase  from  1  to  3  while  the  duration  of  surgery  increased  by  30 
seconds.  IPS  improved  by  34%  (48.7-65.3;  p<0. 00001).  Most  improved  measures  were  a  90.5% 
(31.9-60.8;  p<0.00001)  increase  in  procedural  steps,  a  200%  (21.2-64.1;  p<0.00001)  increase  in 
anatomic  knowledge  and  a  31.4%  (57.3-75.3;  p<0. 00001)  improvement  in  technical  skills.  Post 
ASSET  training,  the  surgical  resident  performance  was  comparable  to  that  of  experts,  three  of 
whom  failed  to  decompress  at  least  1  of  4  compartments  (average  3.3  compartments 
decompressed).  In  comparison  to  post  ASSET-trained  residents,  surgeons  evaluated  2-5  years 
after  training  show  overall  degradation  (IPS:  -3%;  65.3-63.1),  decrease  in  anatomic  knowledge  (- 
15.7%;  64.1-54),  procedural  steps  (-5.9%;  60.8-57.2)  with  fewer  decompressed  compartments 
(mean  2.7)  and  an  increase  in  surgical  time  (+2  min). 

Conclusions:  Anatomical  knowledge  is  a  critical  feature  of  successful  decompression  of  the  four 
compartments  of  the  leg.  Anatomical  knowledge  and  the  ability  to  identify  landmarks  and 
structures  showed  the  highest  amount  of  score  degradation  2-5  years  after  training.  The  1  day 
ASSET  course  was  highly  impactful  for  improving  surgical  skills  to  compensate  for  the 
decreasing  on-the-job  training  during  residency.  These  results  also  show  the  need  for  refreshing 
or  just-in-time  re-training  the  knowledge  and  skills  acquired  from  supplementary  courses. 
Supported  by  W81XWH- 13-2-0028  from  USAMRMC  JCP-1  and  CDMRP. 


Learning  Objectives 

1:  Supplementary  surgical  skills  training  significantly  improves  performance  competency. 

2:  Knowledge  of  anatomical  structures  and  surface  landmarks  are  critical  for  successful  4 
compartment  lower  extremity  fasciotomy. 

3:  Within  5  years  of  training,  competency  in  performing  a  4  compartment  lower  extremity 
fasciotomy  declines.  This  indicates  the  need  for  periodic  skills  refreshing. 
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Appendix  28:  Which  Combat  Surgical  Skills  Degrade  after  Training? 


Stacy  Shackelford1,  Evan  Garofalo2,  Valerie  Shalin3,  Kristy  Pugh4,  Hegang  Chen5,  George 
Hagegeorge4,  Jason  Pasley1,  Sharon  Henry6,  Mark  Bowyer7,  Colin  Mackenzie4. 

1 USAF  Center  for  Sustainment  of  Trauma  and  Readiness  Skills,  Baltimore,  Maryland 
“Department  of  Anatomy  and  Neurobiology,  University  of  Maryland  School  of  Medicine 
"Department  of  Psychology,  Wright  State  University,  Ohio 

4Shock  Trauma  Anesthesiology  Research  Center,  University  of  Maryland  School  of  Medicine 
^Department  of  Epidemiology,  University  of  Maryland  School  of  Medicine 
^"Department  of  Surgery,  University  of  Maryland  School  of  Medicine 

' 7 

The  Norman  M.  Rich  Department  of  Surgery,  Uniformed  Services  University  of  Health 
Sciences 

INTRODUCTION:  Bleeding  is  the  leading  cause  of  early  preventable  death  in  military  and 
civilian  casualties.  Trauma  surgeons  must  maintain  proficiency  in  the  surgical  exposure  and 
control  of  major  blood  vessels.  The  ability  to  objectively  measure  durability  and  decay  of  such 
combat-specific  surgical  skills  is  crucial  to  training  and  maintaining  casualty  care  teams  in  a  state 
of  readiness.  We  hypothesized  measurable  categories  of  combat  surgical  skill  degradation 
occurring  2  or  more  years  after  training  in  the  absence  of  direct  practice. 

METHODS:  We  evaluated  individual  surgeon  performance  within  2  weeks  of  completing  the 
Advanced  Surgical  Skills  for  Exposure  in  Trauma  (ASSET)  course  in  39  fourth  and  fifth  year 
surgical  residents  and  compared  these  surgeons’  performance  to  that  of  10  surgeons  ASSET - 
trained  2-5  years  ago  and  to  8  expert  trauma  surgeons  in  practice  an  average  of  16.1  +/-  10.8 
years.  All  surgeons  were  evaluated  with  a  case-based  scenario  and  evaluation  script  while 
performing  the  same  4  procedures  (axillary,  brachial,  femoral  artery  exposure  and  lower 
extremity  fasciotomy)  on  unpreserved  cadavers.  Two  co-located  trained  evaluators  conducted 
evaluation.  Performance  measure  included  previously  validated  technical  and  non-technical 
skills  metrics  obtained  by  checklists,  and  global  ratings  for  these  4  procedures.  Individual 
procedure  scores  for  each  procedure  and  a  composite  score  for  all  four  procedures,  termed 
trauma  readiness  index  (TRI)  were  compared  between  immediately  post- ASSET  versus  ASSET- 
trained  2-5  years  ago  and  experts.  Skill  subtype  categories  included  overall  knowledge,  anatomic 
knowledge,  patient  management  knowledge,  technical  skills,  and  procedural  steps  to  determine 
possible  retraining  needs. 

RESULTS:  The  performance  of  expert  surgeons  was  significantly  better  than  all  other  groups 
(TRI  expert=68,  p<0.00001).  For  39  residents,  mean  TRI  scores  improved  significantly  after 
ASSET  training  (TRI  pre=46,  TRI  post=61,  p<0. 00001),  and  scores  decreased  slightly  for 
surgeons  ASSET-trained  2-5  years  ago  (TRI=57,  p>0.05).  The  anatomic  knowledge  score  for 
surgeons  trained  3  to  4  years  ago  was  lower  than  residents  after  ASSET  training  (pcO.00001); 


67 


there  was  also  a  trend  for  decreased  procedural  steps  scores  (p>0.02).  Patient  management 
knowledge  and  technical  skills  did  not  decline. 

CONCLUSION:  Relative  to  performance  immediately  after  ASSET  training,  performance  of 
important  combat  surgical  procedures  degraded  significantly  2-5  years  later.  Relative  to  recently 
trained  residents,  practicing  surgeons  who  received  ASSET  training  2-5  years  ago  yielded 
significantly  lower  scores  for  anatomic  knowledge,  with  a  trend  toward  lower  procedural  step 
scores.  This  suggests  that  previously  trained  surgeons  in  active  surgical  practice  would  benefit 
from  a  trauma-specific  skills  refresher  course  that  focuses  on  review  of  anatomy  and  procedure 
specific  steps,  rather  than  patient  management  or  surgical  technical  skills.  Commercial  pilots 
must  pass  semiannual  evaluations  on  full-mission  flight  simulators  to  maintain  licensure. 
Surgeons  have  no  such  operative  skills  re-evaluation  requirement.  This  study  shows  that  re¬ 
training  of  surgeons  is  needed  in  rarely  performed  combat  casualty  surgical  skills  within  a 
minimum  of  two  years,  but  the  optimal  timing  and  method  of  refreshing  these  skills  and 
preventing  degradation  requires  assessment.  Supported  by  W81XWH- 13-2-0028  from 
USAMRMC  JCP-1  and  CDMRP. 
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Appendix  29:  Are  Physical  Model  Assessments  Comparable  to  Cadaver  for  Combat  Surgical 
Technique? 
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Appendix  30:  Perception  Does  Not  Equal  Reality  for  Resident  Vascular  Trauma  Skills 


Mark  W.  Bowyer,  MD,^  Stacy  A.  Shackelford,  MD,  ,  Evan  Garofalo,  PhD, : 

Kristy  Pugh,  MS,d  and  Colin  F.  Mackenzie,  MBChB 

a  The  Norman  M.  Rich  Department  of  Surgery,  Uniformed  Services  University  of  the  Health  Sciences, 

Bethesda,  Maryland 

b  USAF  Center  for  Sustainment  of  Trauma  and  Readiness  Skills,  R  Adams  Cowley  Shock  Trauma  Center,  University 
of  Maryland,  Baltimore,  Maryland 

c  Department  of  Anatomy  and  Neurobiology,  University  of  Maryland  School  of  Medicine,  Baltimore,  Maryland 
d  Shock  Trauma  Anesthesiology  Research  Center,  University  of  Maryland  School  of  Medicine,  Baltimore,  Maryland 

Background:  Experience  with  the  management  of  vascular  trauma  by  senior  surgical  residents  is 
increasingly  limited.  When  queried  about  their  understanding  of  anatomy  and  ability  to  perform 
specific  vascular  exposures,  residents  express  a  moderately  high  level  of  confidence.  We 
hypothesized  that  this  perception  does  not  equal  reality. 

Methods:  A  total  of  42  senior  surgical  residents  participating  in  an  ongoing  validation  study  of 
the  Advanced  Surgical  Skills  for  Exposures  in  Trauma  course  were  asked  to  self-assess  their 
baseline  (pre-course)  confidence  of  their  understanding  of  the  anatomy  required  to  perform  and 
their  ability  to  perform  exposure  and  control  of  the  axillary,  brachial,  and  femoral  arteries,  as 
well  as  lower  extremity  fasciotomy  using  a  5 -point  Likert  scale.  Residents  then  performed  the 
four  procedures  on  a  fresh  cadaver  model  and  were  scored  in  real  time  by  experts  using  a  global 
assessment  of  anatomic  knowledge  and  readiness  to  perform.”  The  Student  t-test  was  used  with  a 
set  at  P  <  0.05. 

Results:  Residents  consistently  rated  their  understanding  of  anatomy  and  their  ability  to  perform 
the  procedures  significantly  higher  than  expert  evaluator  ultimately  scored  them.  Evaluators  also 
deemed  that  residents  would  be  unable  to  perform  without  help  65%e86%  of  the  time. 

Conclusions:  Senior  residents  are  ill -prepared  to  perform  the  procedures  studied  and  have  an 
unwarranted  confidence  in  their  knowledge  and  abilities.  Perception  clearly  does  not  equal 
reality  in  preparing  these  trainees  to  perform  as  advertised.  The  low  global  scores  for  anatomy 
and  performance  should  be  a  wake-up  call  for  surgical  educators  prompting  curricular  reform 
and  evaluation. 
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Background:  Experience  with  the  management  of  vascular  trauma  by  senior  surgical  resi 
dents  is  mcreasngty  b noted  When  queried  about  their  undentending  of  anatomy  and 
ability  to  perform  specific  vascular  exposures,  residents  express  a  moderately  high  level  of 
confidence.  We  hypothesized  that  this  perception  does  not  equal  reality 
Methods  Atotelof  42  senior  sir  gical  residents  participating  in  an  ongoing  validate an  study 
af  five  Advanced  Surgical  Skills  for  Expo  sixes  in  Trauma  course  were  asked  to  self  assess 
then  baseline  (prerow se)  confidence  of  their  understanding  of  the  anatomy  required  to 
perform  and  their  ability  to  perform  exposure  and  control  of  fixe  axillary,  brachial,  and 
femoral  arteries,  as  weS  as  lower  extremity  fasdotomy  unrig  a  S  posit  Likert  scale.  Resi 
dents  then  performed  the  foix  procedures  on  a  freA  cadaver  model  and  were  scored  in 
real  time  by  experts  using  a  global  assessment  of  anaemic  knowledge  and  readiness  to 
perform  *  The  Student  t  test  was  used  with  a  set  at  P  c  OlOS. 

Results  Readente  consistentlyrated thear  indeistandmgaf anatomy  andthearabihtyto  perform 
the  procedures  agnficantly  higher  than  expert  evaluator  ultimately  scared  them  Evaluators 
also  deemed  that  readents  wo  lid  be  unable  to  perform  without  help  65%-tfftaf  the  time 
Conclusions  Senior  readente  are  ill  prepared  to  perform  the  procedures  studied  and  have 
an  unwarranted  confidence  in  their  knowledge  and  abilities  Perception  dearly  does  not 
equal  reality  in  jxe paring  these  trainees  to  perform  as  advertized  The  low  global  scares  for 
anatomy  and  performance  should  be  a  wake  up  call  for  surgical  educators  prompting 
curricular  reform  and  evaluation 
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Appendix  31:  Sensor-Free  Computer-Vision  Hand-Motion  Entropy  and  Video- Analysis  of 
Technical  Performance  during  Open  Surgery:  Proof  of  Concept  Report  of  Methodology  and 
Analysis 


Seasor-free  Computer- Vision  hand-motion  entropy  and  video-analysis  of  technical  performance 
during  open  surgery:  proof  of  concept  report  of  methodology  and  analysis 

Cohn  F  Mackenzie.  *•*  Darcv  Watts. 1  Rajan  Patel. 1  Shhning  Yang,1  ■*  George  Hagegeorge,  1  Evan 
Garo£ak>.;  "  Peter  F  Hu, ' A4,  *  Adam  Puche;*  Kristy  Pugh.  1  Guinevere  Granite,  '*  Lynn  G  Stansbury. 1 
Stacy  Shackelford,*  w  Samuel  Tisherman  *, 5  Valerie  Shalin; 

'Shock  Trauma  Anesthesiology  Research  Center;  'University  of  Maryland  School  of  Medicine 
Departments  of  Anatomy  and  Neurobiologv.  ’Anesthesiology,  and  *  Surgery.  5  R  Adams  Cowley  Shock 
Trauma  Center,  University  of  Maryland  Medical  Center;  °  University  of  Arizona .  Pep  aliment  of 
Anatomy,  Phoenix  Arizona.  1  Wright  State  University  Department  of  Psychology,  Dayton  Ohio  *  US 
Airforce  Center  for  die  Sustainment  of  Trauma  and  Readiness  Skills,  Baltimore 


Domains  such  as  piloting,  driving  or  laparoscopic  surgery'  support  the  collection  and  analysis  of  motor 
control  data  because  hitman  action  is  executed  through  instruments  However,  domains  lacking  such 
instrumentation,  such  as  conventional  surgery,  cannot  exploit  instrument -dependent  data  collection 
methods  Compensatory  approaches  enslaving  hand-motion  sensors  risk  task  interference  and  complicate 
performance  analysts  Standard  observational  judgment  is  cosily  and  logisncally  challenging  We  employ 
computer  vision  (ex'1)  hand  motion  analysis,  video  and  Shannon  Joint  Entropy  analysis  to  demonstrate  the 
potential  for  a  partially  automated  evaluation  of  surgical  technical  performance  Color  coded  gloves  and 
marked  surgical  hand-tools  support  a  video-based  approach  to  performance  evaluation.  We  demonstrate 
the  promise  of  this  approach  for  discriminating  between  expert  trauma  surgeons,  surgical  residents  and 
anatomists  performing  axillary  artery  exposure  and  control  (AA)  an  fresh  cadavers,  and  correlate  die  results 
with  an  independently  validated  observational  approach  (IPS)  to  surgical  skill  evaluation.  Hand  motion 
analysis  techniques,  which  w  ere  congruent  with  IPS  evaluations  can  discriminate  levels  of  surgical  skill 


Background: 

Skilled  motor  behavior  is  key  to  task  success  in  many 
domains,  such  as  avianon  and  driving.  But  m  these  example 
domains  instruments  mediate  behavior,  tar  ilirsfTn g  the 
collection  and  analysis  of  motor  behavior  In  contrast,  surgical 
skill  is  not  necessarily  executed  using  instruments  that 
facilitate  data  collection  This  challenges  the  evaluation  of 
surgical  competence,  for  example  of  surgical  residents  Time- 
consuming  observational  methods  are  iogtsticilly  challenging, 
requiring  the  co-location  of  evaluators  and  candidates,  and 
may  include  evaluator  biases.  Efforts  have  been  made  to 
standardize  surgical  trainee  evaluations  and  eliminate  bias 
(Martin  et  al.  1997)  However,  these  observation -based 
approaches  still  suffer  horn  logistical  challenge. 

Hand-motion  analysis  has  potential  as  an  unbiased, 
accurate,  and  cost-effective  means  to  evaluate  surgeon 
technical  performance  The  elements  of  manual  dexterity  on 
which  surgical  skill  depends  have  been  increasingly  well 
documented  over  the  last  decade  and  are  related  to  levels  of 
experience  (Ahmidi  et  aL,  2015;  D'Angelo  et  al.,  2015;  Datta 
et  al.,  2001;  Daria  et  al.,  2006;  Overby  &  Watson.  2014). 
However,  these  studies  rely  on  synthetic  models  or  partial 
tasks  to  simplify  the  hand  morion  analysis  (Watson,  2014)  or 
focus  on  endoscopic  or  specialized  environments  (Ahmidi  et 
al.,  2015;  Watson,  2014)  Few  studies  evaluate  open  surgical 
procedures  because  captunng  band  movement  m  this  setting  is 
more  complex.  Open  surgical  procedures  vary  widely, 
requiring  assessment  methods  that  allow  for  freedom  of  hand 
and  instrument  movement.  Ideally,  these  methods  should  be 
sensor-free  to  avoid  interference  with  hand  motion  and 
surgical  performance 

The  combination  of  kinematic  data  collection  and 
analysis,  and  video  surgical  gesture-recognition  has  potential 
to  address  these  requirements,  but  exploration  has  been  limited 


to  date  to  laparoscopic  procedures  (Dos is  et  aL,  2005). 
Dynamic  systems  theory  of  motor  development  emphasizes  a 
redaction  m  variability  as  part  of  the  learning  process;  optimal 
movement  variability  balances  the  benefits  of  rigid  control  and 
randomness  of  movement,  thus,  a  complexity  measurement 
might  be  expected  to  differ  during  training.  (Mackay.  2003) 
Shannon  Joint  Entropy  has  been  used  to  summarize  the 
systematic  information  conveyed  by'  bimanual  hand 
movements,  using  computer  vision  (CV)  algorithms  derived 
from  measurement  of  frequency,  direction,  and  speed  of 
movement  changes.  (Watson.  2012;  Mackay,  2003)  Entropy 
should  decrease  with  hand  motion  efficiency  and  has  been 
used  to  analyze  hand  movements  in  training  models  of 
suturing  (Watson.  2012)  but  not  in  open  surgery 

We  conduct  entropy  analysis  of  surgical  skill  with 
modest  modifications  of  die  task.  A  panel  of  tranma  experts 
was  consulted  for  assistance  in  die  development  of  the 
procedural  analyses  as  previously  described  (Mackenzie  et  al  , 
2015 ;  Shackelford  et  aL,  2015)  to  provide  metrics  that  might 
differentiate  between  novice  and  experienced  surgeons  Other 
possible  hand-motion  metrics  were  obtained  from  review  of 
the  existing  literature  (Ahmidi  et  aL,  2015;  Overby  &  Watson, 
2014).  Different  glove  colors  for  each  hand  support  die  use  of 
CV  algorithms  without  requiring  special  hand  sensors.  Our 
environment  provides  numerous  measures  of  skilled  motor 
behavior,  including  smoothness  of  movement,  velocity  and 
acceleration,  path  length  of  the  hand  idle  time,  and  bimanual 
dexterity.  To  examine  the  variability  of  measures  with  respect 
to  expertise,  we  tested  participants  with  likely  differences  m 
skill:  attending  trauma  surgeons,  surgical  residents  before  and 
after  training  m  a  specific  procedure,  and  PtiD  anatomists  (as 
a  Soto  gate  for  anatomically  knowledgeable,  but  inexperienced 
clinical  surgery  operators)  Furthermore,  we  associated  die 
measures  of  skilled  motor  behavior  with  our  own  validated 
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Appendix  32:  Trauma  Training  Courses  for  Surgeons  and  Validation  of  their  Efficacy 

Objective:  Review  the  existing  literature  on  trauma  training  courses.  Summarize  currently 
available  data  on  validation  of  trauma  surgical  skills  training  and  course  benefits. 

Design:  Literature  search  of  Pubmed  systematic  review  database  was  conducted  identifying 
systematic  reviews  of  trauma  training  courses. 

Setting:  Shock  Trauma  Anesthesiology  Research  Center,  University  of  Maryland  School  of 
Medicine,  US  Airforce  Center  for  the  Sustainment  of  Trauma  and  Readiness  Skills,  Baltimore, 
Uniformed  Services  University  of  Health  Sciences,  Bethesda,  Maryland. 

Results:  Multiple  surgeon  trauma  training  courses  were  found  that  were  evaluated  with  self- 
reported  confidence  or  in  3  courses  with  subjective  ratings.  The  benefits  of  the  Advanced 
Surgical  Skills  for  Exposure  in  Trauma  (ASSET)  course  have  undergone  preliminary  validation 
with,  objective  ratings,  checklists,  global  and  self-reported  confidence  evaluations.  No  objective 
skill-durability  data  have  been  published  for  hands-on  cadaver  or  live-tissue  hemorrhage  control 
courses.  Introductory  surgical  boot  camps  for  new  surgical  residents  teaching  procedural  skills 
are  correlated  with  In-Training  exam  scores  and  assist  cumulative  resident  evaluations.  There  are 
no  controlled  clinical  trials  of  Advanced  Trauma  Life-Support  (ATLS)  showing  that  training 
changes  trauma  management  outcomes  or  mortality.  There  are  studies  showing  that  ATLS 
improves  organizational  and  priority  approaches  and  clinical  skills  for  management  of  multiple 
trauma  patients. 

Conclusions:  There  are  a  large  number  of  different  trauma  skills  -training  courses.  Very  few 
have  been  validated  to  show  benefit  with  objective  metrics.  Large  variations  in  duration, 
resource  requirements  and  cost  suggest  benefits  from  standardization  of  the  trauma  surgical 
training  component  of  these  courses.  A  trauma  surgical  performance  benchmark  is  essential  and 
needed  to  measure  the  adequacy  and  quantitate  benefits  of  surgical  training  and  competence  of 
trauma  surgeons. 
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Appendix  33:  Performance  of  Combat  Surgical  Skills  before  and  after  ASSET  training 
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Appendix  34:  IRB  Approval  Letter  2015 


University  of  Maryland.  Baltimore 
Institutional  Review  Board  (IRB) 
Phone:  (410)706-5037 
Fax:  (410)  706-4189 
Email:  hrpoasoni  umarvland.edu 


APPROVAL  OF  RESEARCH  NOTIFICATION 


Date:  August  4.  2015 

To:  Cohn  Mackenzie 
RE:  HM-HP-00054443-6 
Protocol  Version  and  ID  #:  N/A 
Type  of  Submission.  Modification 
Type  of  IRB  Review.  Expedited 
Modification  request  dated:  7/22/2015 

Modification  Approval  Date:  8/4/2015 
Approval  for  this  project  is  valid  until  11/16/2015 


This  is  to  certify  that  the  University  of  Maryland.  Baltimore  (UMB)  Institutional  Review  Board  (IRB)  approved  die 
above  referenced  modification  request  for  the  protocol  entitled.  "Use  of  performance  .measure:  to  e\aluate. 
document  competence  and  deterioration  of  ASSET  surgical  skitter. 

The  IRB  approved  this  modification  via  expedited  review  pursuant  to  Federal  regulations  45  CFR  46. 1 10(b)(2)/21 
CFR  56.110(b)(2). 

The  IRB  made  the  following  determinations  regarding  this  submission: 

-  Written  informed  consent  is  required  Only  the  valid  IRB- approved  informed  consent  forzn(s)  in  CICERO  can  be 
used 


Below  is  a  list  of  the  documents  attached  to  your  application  dial  have  been  approved: 
Eligibility  Checklist  for  HP-00054443_3  vl-10-2014-1389391 163858 
ASSET  proposal 
Stud}*  Timeline 

Uniformed  Services  University  of  Health  Samces 

JSURG_1155  Proof.pdf 

Script  of  Interview,  docx 

Patel  CITI  certification  2015.pdf 

Robinson  Citi  certificate.pdf 

Robinson  Citi  certificate.pdf 

Watt5_Cin  certificate  pdf 

Teeter_cmCompletionRepon  1 787049_2  pdf 

usheiman  cm  courses  l-12-15.pdf 

Granite  CTTLpdf 


In  conducting  this  research  you  are  required  to  follow  the  requirements  listed  in  the  INVESTIGATOR  MANUAL. 
Investigators  are  reminded  that  the  IRB  must  be  notified  of  any  changes  m  the  study.  In  addition,  the  PI  is 
responsible  for  ensuring  prompt  reporting  to  the  IRB  of  proposed  changes  in  a  research  activity,  and  for  ensuring 
that  such  changes  in  approved  research,  during  the  period  for  which  IRB  approval  has  ahead}'  been  given,  may  not 
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Appendix  35:  Request  for  Statement  of  Work  Modifications  September  2015 


Ob 

8  j  J  UN  I  VERS  IT  Y  of  M  A  ilY  i .  A  ND 

JJi  SCI  K'J'H  OF  Vi F  D -C(NL 

***  -hAJMA  A\U  ^NKSfkFriCI  OCv 

nr.=:F.j.rri  r.r k- 

:•  **'.*  vc:.  •  i.  ►  l._  ci 

LriMra*.  kU 

+  t)  T-*  j,\:>  :  .1 I :j::  lyi* .  _|_| 

vv-  *‘:**v.  V’**'  vx>j 


COLIN  F.^ACK&N?!? 
ProlcSsof  Arrjsttaplolo-?:,' 
Associate  ?n>7«s»o'  -’tTv.-.ioloay 


Sepl  ember  20)5 

I  rcqncsl  eonxidenilTcri  of  Hi roe  c h ko  ggn.'Tn Dili  fic-ali i ins  lo  ihe  Statement  of  Work  i’SOWI  for 
W81XWH-i3-MM)8 

"l:\e  (if  {’erfonitCHCd  Measures  to  Evdiutt e.  Oocuniertt  Com/reliince  and  DeSerinraH t:n  cj 

/itivuncctl  Surgical  SxiUs  lixytMure  fi.r  Trauma  CASSK!']  Surgical  Skiifx".  The  Tl  lie  v.ns 

abbreviated  as  Kereituon  ax.  Assessment  of  Surgical  Performance  (RASP) 

Principal  Investigator:  l)r.  Macktittie,  U  nicer!!  it  v  til'  Maryland.  rtaltiniorc  (UMR). 

A.  Request  npprovflJ  tor  3  Changes:  Modifications  to  the  original  SOW 

J )  Add  an  additional  surgical  procedure  ;o  avoir:  ‘learning  bias".  For  the  dO  surgeons  whose 
peril M  inlTiOe  we  have  ev illumed  hcMiire  and  within  2  weeks  of  A^FT  training  (sec  paper 
altaehed.l  vve  are  lilrthei  evdunlini;  them  fas  pel  original  SOW)  !2  or  <8  months  iiflsr 
training  to  assess  skill  retention.  However.  there  is  a  “traiiii-ia  bias”  with  this  nielli  ixlolugv  in 
that  the  surgeon  gains  knowledge  toe.  in  proves  performance  because  of  repeated  lesbnu.  <m 
tlic  samed  irroccdtu'cs  and  the  debrief  that  occtua  each  rime.  In  the  requested  rtmdiJi  cation  x > 
Ihc  SOW.  xvc  will  add  evaluation  of  an  adHiiioolfurlhcr  procedure,  carotid  orrery  exposure 
(CAD)  and  conitui  of  common  itueuwl  and  external  euro  lie  an-ury.  .0  lire  12  or  US  mo  nth 
evaluations.  llcuaune  this  procedure  was  taught  <m  Ihc  ASS  FT  course  bur  net  evaluated  either 
lielivre  or  alter  training.  Inis  evaiualioii  will  be  a  true  "skill  degradation"  since  ASSET 
Training.  We  do  also  enliecl  surgical  case  logs  I’rmr  each  surgeon,  so  we  UIH  determine  if 
the  surc,eov  performed  any  carotid  expostue procedures  in  the  inlervul  beLvvuoi? peel  - 
ASSHi  training  evaluation  and  skill  retention  evaluation  performed  12  or !  8  months  later. 
There  is  no  cost  for  this  modification. 

2)  Add  evaluation  ofn  cohort  of  expert  pixel  icing  lm;nn  ailont&ng  surgeons.  Wc  developed  tile 
scripl  with  input  from  expert  practicing  Iruuma  surgeons  hul  .vv  have  no  experts  who  vverc 
evaluated  rising  the  same  slandardi^d  scripl  while  they  pcrlbm'  the  same  procedures  as  all 


DfcM  SIHV  .Mi  •  '-1POICINE  ’  N  jfti'VO  -  Fh.M>M«:v  *  MDC.  A*  WORK  •  ORit'JA'i  £  UOirr 
r.vaa;-  HMI  me  il>:< ••  «:  <yi  li-ure'  .U-  Uiuremty  ot«»ryl=nd  of  Muluiliir  •  .V'jriia.i  oars!  r'lhllr  ■n*/  .....  ^divo1.  i3urd*r.  'n  '.Her. 
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Appendix  36:  Request  for  Budget  Modification  September  2015 


1 1  fi  UN  [  VERS11  Yci/'.M  ARY)  A  NO 
NJ  StriKlOi  OF  MFOldNl 

t-iOU;  VJMAAMJ  tNFSTHrFini  OCV 

Rr"rj-rci 


LVL\K  " 

PYOl^iSC-.'  jXrc-Jtheslob  yj* 
*s‘oUaite  Prar^tiut  PK^u>kKry 

II  :w.-  t-WM-.-r.  fi.  r 

•i'OA.’f  irs.%  i  .r*»  i#t  •?:!:  •'<  i  .*ks  ».*:•*  sm**  _u 

r"  «:  a':  .  coj 


September  2015 

I  request  uoRsidenairm  el  jjiodiFeaiion;  to  the  Budget  for  W81XWH'l3>2-i)iHl 

"fixe  nf  Pftrjtmranrit  .Wsv/v:ir.<;.v  fn  Kv/sluuif.,  Docvwm  Cnmpgttaxv  (ma  UeTeJ'iop<itioJ!  of 
Advanced  fSorgiouI SkiUi  ExpotHrafor  Trauma  (ASS K7>  SurgietA  Skills".  The  Title  was 
abb'viviated  ns  Retention  and  Assessment  :il  Surg:cai  Perlbrrnur.ee  (RASP; 

Principal  Investigator:  Ur.  Onfm  Matlifn/ic,  University  of  Maryland,  Baltimore  (UMB). 

Request  fnr  Modifications  to  the  Budget 


Peirmnn::! 

1)  Recruitment  and  biring  of  rejikucnien'.  for  Surgeon  COL  Stscv  Stacker  ford  wiu 
potential  candidate  Dt  Bili  Teeier,  Dr  T ucter  .  u  surgeon  at  the  University  of 
Maryland  v/LIL  be  trained  to  assist  with  video  analysis  art!  help  with  conduct  efthe 
procedures  in  the  Anatomy  Board  and  eval  nate  enrolled  s nru.eun v .  COT.  Shackelford 
will  take  up  duties  at  Institute  Surgical  Research  in  San  Antonin  l'x  or  July  I  si. 

Bed  gel  for  including  Or  Teeter  ar  25  S-o  effort  July  T1  tt>  the  end  of  the  project 
February  ,*4-h  20  Id  to  replace  COL  Stacy  Sl'.ackcSrbicl .  Total  cost  between  Inly  i, 
201 5  fct:  February  14.  2016  al  25  Js  for  Dr.  Tecta  =  SIC.  H32  (Salary  $R266  Fringe 
S2,066j 

2}  Recruitment  and  bump  of  replacement  Tor  Tv,  an  fiat,:  lain  PhD  anatomist  with 
potential  candidate  Guinevere  Granite  PhD  anatomist  On  June  I5(h  2015,  J> 
Garortile  (Research  Coordinator  and  Co-in  vest  ipatori  moved  t<>  a  Faouliy  position  m 
Uni  veruily  cf  Arizona  School  of  Medicine  Department  of  Anatomy  atnriinc  July  1*. 
No  Cost  lo  support  an  Ana  I  twist  Ip  replace  Ur  GarofaJo 

3)  increased  effort  o>r  Prmcjp-f  Investigator  dxing  transition  when  COL  Shackelford 
tell  because  request  lo  obtain  udminki.mlivc  support  was  not  approved.  The  ITincLual 
investigator  has  bad  to  take  on  a  ltd  of  unexpected  adiFtionnJ  work  clfbrt  dtiiii'G  this 
personnel  transition  inducing:  a)  Frticessiisg  ol'lRR  renewals  lo  make  rutnicsls  for 
addition  of  cvaiuatcus  aita  investigators  to  replace  those  leaving  and  in  add  their 
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Appendix  37:  Office  of  Naval  Research  Modified  Pre-Proposal  “Autonomous  Analysis  for 
Technical  Performance  of  Combat  Surgical  Skills” 


WHITE  PAPER 

BAA  Number  ONRBAA15-001 

Project  Tide:  Autonomous  Analysis  for  Technical  Performance  of  Combat  Surgical  Skills. 

Principal  Investigator  (PI):  Adam  Puche  PhD  (Co-PI's)  Valerie  Shalin  PhD,  Shiming  Yang  PhD 

Co-Investigators:  Sam  Tisherman  MD  FACS.  Col.  Stacy  Shackelford  \ID  FACS,  Maj  Jason 
Pasley  DO  FACS,  Mark  Bowyer  MD  FACS,  Hegang  Chen  PhD,  Peter  Hu  PhD.  Consultants: 
Colin  F  Mackenzie  MD,  Hick  Sevdalis  PhD. 

Contact: 

Adam  C.  Puche.  PhD. 

Associate  Professor 

Dept  Anatomy  &  Neurobiology 
University  of  Maryland  School  of  Medicine 
20  Penn  St.  Rm..  251  (mailing) 

685  West  Baltimore  St,  Rm.  280M  (office) 

Baltimore,  Maryland.  21201 
Email:  apuche  u  som  nmarvlancLedu 
Ph.  (410)  706  3530 
Fax  (410)  706  2512 
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Appendix  38:  Full  Proposal  for  BA150077  -  “Refreshing  Combat  Surgical  Skills” 


TECHNICAL  ABSTRACT:  Refreshing  Combat  Surgical  Skills 

Background:  Bleeding  remains  the  commonest  cause  of  preventable  combat  casualty  death,  so  trauma 
surgeons  must  maintain  proficiency  in  surgical  exposure  and  control  of  major  blood  vessels.  Equally,  combat 
medics  must  be  able  to  recall  and  use  life-saving  skills  The  Advanced  Surgical  Skills  Exposures  for  Trauma 
(ASSET)  course  was  developed  to  provide  training  m  exposure  and  control  of  major  vascular  trauma,  however, 
little  data  exists  on  how  durable  this  training  is  or  on  optimal  intervals,  methodology,  or  cost'benefit  ratios  of 
refresher  training.  We  propose  two  related  parallel  studies  to  test  refresher  training  to  achieve  best  performance 
with  least  cost  for:  1)  surgeons  ASSET-trained  2-6  years  ago  and  2)  medical  students  (as  combat  medic 
surrogates)  trained  two  years  previously  in  selected  relevant  techniques.  We  will  compare  the  performance  and 
cost’benefit  of  a)  interval  cadavenc  practice  or  video  mental  rehearsal  to  b)  just-m-time  assistance  via  a  Heads- 
up-Display  of  procedural-step  video  clips  (HUD)  or  tele-mentoring  (TM)  to  c)  no  refresher  training. 

Hypotheses:  1)  significant  degradation  m  performance  of  vascular  control  surgical  procedures  occurs  with  no 
refresher,  2)  periodic  mental  rehearsal  and  ' or  just-m-time  refresher  assistance  is  as  effective  as  cadaveric 
practice;  3)  just-in-time  refresher  methods  like  HUD  or  expert  real-time  TM  improve  surgical  performance  m 
both  surgeons  and  combat  medic  surrogates;  and  4)  HUD  will  have  the  best  cost'benefit  ratio  of  all  of  the  skills- 
refresher  methods  tested.  STUDY  1  tests  which  of  the  refresher  methods  gives  best  cost'benefit  with  respect  to 
multiple  criteria  such  as  time,  equipment,  site  presence,  infrastructure  requirements,  materials,  etc,  for  ASSET 
framed  surgeons.  Aim  1.1  of  this  study'  compares  two  “fixed-time  interval”  refresher  methods,  video  mental 
rehearsal  or  deliberate  cadaveric  practice.  Aim  1.2  compare  two  ‘just-in-time”  refresher  approaches,  HUD  or 
TM.  STUDY  2  tests  whether  Study  1  refresher  methods  can  generalize  to  other  combat  trauma  care  providers 
Aim  2)  evaluates  STUDY  1  refresher  methods  in  a  study  group  of  medical  students  used  as  surrogates  for 
combat  medics  trained  m  femoral  artery  exposure  Finally',  Aim  3)  evaluates  which  of  “fixed-time  interval”  or 
“just-m-time”  refreshing  skills  produces  the  best  surgeon  individual  procedure  scores  (IPS),  also  improves  IPS 
m  medical  student  medic  surrogates,  and  documents  competence  m  comparison  to  historic  IPS  data  on  experts 
and  just-trained  surgeons  from  our  previous  study. 

Design  &  methods:  STUDY  1:  60  surgeons  trained  in  4  ASSET  procedures  2-6  years  ago  will  have  one- 
year-interval  cadaveric  practice  (n=15)  or  video  mental  rehearsal  (n=  15)  versus  just-m-time  assist  with  HUD 
(n=15)  or  TM  (n  =15)  without  interval  refresher  intervention,  all  followed  by'  a  multiple  choice  question  (MCQ) 
test  We  will  test  3  of  the  4  refreshed  procedures  and  add  a  fourth  ‘surprise’  ASSET  procedure,  unrehearsed, 
unpracticed  or  unassisted,  to  determine  degradation  with  no  refresher  intervention.  STUDY  2:  40  Is*  year 
medical  students  will  be  trained  in  a  single  ASSET  procedure  on  a  cadaver,  after  video  mental  rehearsal  (in 
2*1  year,  n=10),  all  forty'  will  be  assessed  m  third  year  on  this  procedure  on  a  cadaver,  using  HUD  or  TM  assist 
or  no  intervalrehearsal  assist  (n  =  10  each)  followed  by  the  MCQ  test,  having  had  (like  combat  medics)  no 
interval  exposure  to  surgery  or  practice  after  training  Using  combinations  of  STUDIES  1  &  2  we  will  compare 
quality  of  performance  outcomes  by'  refresher  method  to  determine  if  low-cost,  just-in-time  assistance  can 
compensate  for  fixed-time  interval  practice  fehearsal  of  surgeons  or  combat  medic  surrogates . 

Data  Collection  and  analysis:  Two  framed  co-located  evaluators  will  conduct  evaluations  on  a  mobile  wireless 
custom  App  using  a  standardized  script,  validated  IPS  metrics  and  audio /visual  recording  of  all  procedures. 
Testing  includes  evaluating  non-technical  and  technical  skills  performance,  individual  checklists  as  well  as 
validated  IPS  metrics.  Global  Performance  Ratings  and  audiovisual  recording  of  all  procedures.  Inter-evaluator 
reliability  will  be  assessed  with  video  review  and  blinding  to  refresher  method  using  infra-class  correlation 
coefficients  (ICC)  (ICOO.7  =  acceptable:  >0.8  =  good;  >0.9  =  excellent)  Statistical  tests  for  IPS  differences 
include  chi-squared  for  categorical  data,  t-tests.  generalized  estimating  equations  and  analysis  of  covariance  to 
adjust  for  confounds  such  as  experience. 

Relevance:  Little  data  currently  exist  on  methods  to  minimize  surgical  skills  degradation.  This  work  will 
provide  objective  measurement  of  the  durability'  of  combat-injuiy-relevant  surgical  skills  and  important 
cost'benefit  data  for  maintaining  combat  care  readiness  The  skills  refresher  methods  anticipated  from  these 
studies  will  be  readily  deployable,  not  require  cadavers,  and  be  suitable  for  both  surgeons  and  combat  medics. 
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Appendix  39:  Pre-Proposal  BA150808  -  “Emergency  Refreshing  of  Combat  Surgical  Skills” 


ARMY  Pre  Proposal  Sept  30*  2015 

Project  Tide:  Emergency  Refreshing  of  Combat  Surgical  Skills. 

Principal  Investigator  (PI/:  Sam  Tisherman  MD  FACS. 

Co-PI's:  Adam  Puche,  PhD  Maj  Jason  Pasle),FACS  COL  Stac y  Shackelford  FACS 

Co-Investigators:  Mark  Bovvyer  FACS ,  Sharon  Henn,  FACS  Valerie  Shalin,PhD  Shimimg 
Tang,  PhD  Hegang  Chen,  PhD  Peter  Hu  PhD 

Consultants:  Colin  F  Mackenzie  MD,  Prof  Nick  Sevdalis  PhD 

Problem  to  Be  Studied: 

Commercial  pilots  must  pass  semiannual  evaluations  on  full-mission  flight  simulators  to 
maintain  then  licensure  Surgeons  have  no  such  operative  skills  re-evaluation  requirement,  and 
the  causes  and  prevention  of  surgical  skills  degradation  remain  largely  unexplored  Because 
Chilian  surgical  practice  prorides  limited  opportunities  for  management  of  combat-like  injuries, 
cadaveric  courses  have  been  developed  to  tram  military  and  civilian  trauma  surgeons  m  combat 
surgical  skills.  The  Advanced  Surgical  Skills  Exposures  for  Trauma  (ASSET)  training  program, 
developed  by  the  American  College  of  Surgeons  Committee  on  Trauma  and  adopted  by  military 
pre -deployment  training  platforms,  teaches  surgeons  how  to  expose  and  control  major  vascular 
trauma.  An  ongoing  aspect  of  the  program  is  evaluation  of  skills  deterioration  over  time  These 
evaluations  include  a  need  to  find  the  best  approach  to  refreshing  ASSET  skills 

Surgeons  deploying  to  care  for  combat  casualties  must  be  proficient  m  the  exposure  and  control 
of  major  vascular  injuries  and  m  fasciotomy  No  current  standards  exist  for  assessing  basic 
competency' m  or  retention  of  these  skills  after  training  or  the  surgeon’s  competency  m 
preparation  to  care  for  combat-associated  injury.  Likewise,  there  is  no  current  mechanism  for 
assessing  the  durability'  or  factors  in  degradation  of  these  skills,  particularly  after  return  to 
peacetime  practice,  where  these  skills  are  unlikely  to  be  used  with  any  frequency.  We  propose  to 
test  two  approaches  to  refresh  skills  in  a  cohort  of  surgeons  3  years  and  more  after  ASSET 
Training  when  we  know  that  the  majority  of  surgeons  have  skill  degradation  as  a  result  of 
standardized  testing  using  unpreserved  cadavers  m  a  previously  funded  study. 

Background:  As  a  result  of  US  Army  funding  (W81XWH-1 3-2-0028)  Use  of  Performance 
Measures  to  Evaluate,  Document  Competence  and  Deterioration  of  Advanced  Surgical  Skills 
Exposure  for  Trauma  (ASSET)  Surgical  Skills  )  we  developed  and  validated  surgeon 
performance  metrics  for  evaluating  technical  and  non-technical  surgical  skills  including  an 
individual  procedure  score  (IPS)  for  performance  of  4  ASSET  procedures  on  unpreserved 
cadavers  — exposure  of  axillary',  brachial,  and  femoral  arteries  and  fasciotomy[6] — included 
among  the  57  procedures  taught  on  the  1  day  ASSET  course  The  IPS  metrics  mclude  individual 
checklists  as  well  as  validated  IPS  Likert  scales  for  surgical  technical  skills  metrics  and  error 
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Appendix  40:  Pre-Proposal  BA150077  -  “Refreshing  Combat  Surgical  Skills” 


Project  Title:  Refreshing  Combat  Surgical  Skills  for  Vascular  Control. 

Principal  Investigator  (PI):  Colin  F  Mackenzie. 

Co-PI:  Stac y  Shackelford, 

Co-Investigators:  Mark  Bowy  er,  Sharon  Henry,  Evan  Garofalo,  Valerie  Shalin,  Peter  Hu, 
Shimimg  Tang,  Jason  Pasley 

Problem  to  Be  Studied: 

Commercial  pilots  must  pass  semiannual  evaluations  on  full-mission  flight  simulators  to 
maintain  their  licensure.  Surgeons  have  no  such  operative  skills  re-evaluation  requirement,  and 
the  causes  and  prevention  of  surgical  skills  degradation  remain  largely  unexplored.  Because 
civilian  surgical  practice  provides  limited  opportunities  for  management  of  combat-like  mjunes. 
cadaveric  courses  have  been  developed  to  tram  military  and  civilian  trauma  surgeons  m  combat 
surgical  skills.  The  Advanced  Surgical  Skills  Exposures  for  Trauma  (ASSET)  training  program, 
developed  by  the  American  College  of  Surgeons  Committee  on  Trauma  and  adopted  by  military 
pre -deployment  training  platforms,  teaches  surgeons  how  to  expose  and  control  major  vascular 
trauma.  An  ongoing  aspect  of  the  program  is  evaluation  of  skills  retention  over  tune.  These 
evaluations  including  different  approaches  to  refreshing  ASSET  skills  versus  no  refreshing. 

A  range  of  professional,  technical,  and  social  factors  have  also  decreased  trauma  surgery 
experience  for  residents  including  factors  that  affect  inner  city  violence,  non-surgical  options  for 
treatment  of  solid  organ  mjunes.  especially  interventional  radiological  advances,  and  shortened 
residency  training  hours.  All  have  about  halved  trauma  surgery  caseloads  and  experience  for 
graduating  chief  residents  in  die  last  decade,  from  60.4  cases  in  1999,  to  33.5  cases  in  2012.  [1] 
This  diminished  operative  expenence  in  trauma  is  especially  true  for  those  surgical  skills  most 
relevant  to  caring  for  vascular  trauma  [2,3,4],  Over  the  last  ten  years  the  mean  number  of  major 
vascular  repairs  for  trauma  (including  repair  of  thoracic  vessels  abdominal  aorta,  penpheral  and 
other  vascular  mjunes)  done  over  the  course  of  a  residency  m  general  surgery  reported  by 
graduating  chief  residents  to  the  Amencan  Board  of  Surgery  decreased  from  average  5.0  m 
2001-2002  to  2.1  m2010-2011  [1,4].  Significant  numbers  of  trainees  have  no  expenence  with 
canng  for  major  vascular  trauma  The  av  erage  number  of  fasciotomies  reported  was  1.2.  The 
average  number  of  exposures  of  the  brachial  artery  reported  by  chief  residents  in  2010-201 1  was 
0.0.  This  training  gap  has  significant  consequences.  Inexpenence  with  lower  extremity 
fasciotomy  has  contributed  to  loss  of  life  and  limb  among  casualties  m  the  cuirent  conflicts  due 
to  incomplete  or  delayed  fasciotomies.  [5] 

Surgeons  deploying  to  care  for  combat  casualties  must  be  proficient  m  the  exposure  and  control 
of  major  vascular  injuries  and  m  fasciotomy.  No  cuirent  standards  exist  for  assessing  basic 
competency  m  or  retention  of  these  skills  after  training  or  the  surgeon’s  competency  in 
preparation  to  care  for  combat-associated  injury.  Likewise,  there  is  no  current  mechanism  for 
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Appendix  41:  Procedure  Case  Scripts  Used  During  Surgeon  Evaluations 


Date: 

(Circle  timing):  Pre  Post 

1st  Trial 

Circle  type  of  trial:  Cadaver  /  Model 


Name  of  Evaluator: 
Name  of  Candidate: 


Case  One:  Axillary  Artery 

Case  Presentation: 

•  You  are  called  to  the  Emergency  Department  to  see  a  24  y/o  male  who 
was  shot  during  an  attempted  robbery  sustaining  a  single  gunshot  wound 
to  the  upper  anterior  lateral  Right/Left  Chest. 

•  He  was  reported  to  have  a  large  amount  of  bright  red  blood  at  the  scene, 
but  is  currently  not  bleeding. 

•  He  is  complaining  of  pain  at  the  site  of  the  wound  and  inability  to  move 
his  arm. 

[Advance  slide  to  show  image  of  wound] 

[Advance  slide  to  continue  narrative] 

•  He  is  awake  and  talking  with  bilateral  and  equal  breath  sounds  and  a  BP 
of  80/60  and  a  heart  rate  of  130  after  2  liters  of  lactated  ringers 

•  There  is  a  single  wound  as  seen  with  no  other  obvious  trauma  and  no 
"exit  wound".  His  hand  is  cool  and  pale. 


l 
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Appendix  42:  RASP  Evaluator  Training  Handbook  &  Video 


RASP  Evaluator  Training  Handbook 
December  2013 


Using  only  small  portion  of  incision 
space  and  "Keyhole"  Surgery 
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Appendix  43:  ASSET  Consent  Form 


UNIVFKSITYrj/MARYI  AND 
1  Hfc  FOUNDING  <1  AMI’ Li 


RESEARCH  CONSENT  FORM 

Protocol  Title:  Use  of  performance  measures  to  evaluate,  document 
competence  and  deterioration  of  ASSET  surgical  skills:  ASSET  Study 

Study  No.:  HP-00054443 

Principal  Investigator:  Colin  Mackenzie  MD.  410-328-7488 
Co  Investigators:  Mark  Bows  er  MD,  Stacy  Shackelford  MD,  Sharon  Henry  MD 

Sponsor:  Department  of  Defense  /  TATRC 


You  are  being  asked  to  participate  in  a  research  stud)'  Your  participation  in  this  stud)'  is 
voluntary  and  you  may  ask  questions  at  any  time  Before  you  decide,  it  is  important  for  you  to 
understand  why  the  research  is  being  done  and  what  it  will  involve.  Please  take  time  to  read  the 
following  information  carefully  You  may  keep  a  copy  of  this  consent  form  to  think  about  before 
making  your  decision 


PURPOSE  OF  STUDY 

The  purpose  of  the  study  is  to  assess  the  Advanced  Surgical  Skills  Exposures  for  Trauma 
(ASSET)  course  and  ASSET  skills  retention  for  1-  5  years  after  training. 

You  are  being  asked  to  participate  m  this  ASSET  Stud)’  either  as  an  expert  a  novice  (never 
participated  in  ASSET  training)  or  as  a  Senior  surgeon  who  has  previously  taken  a  full  ASSET 
course.  If  you  choose  not  to  participate,  there  will  be  no  loss  of  benefits  to  which  you  are 
otherwise  entitled. 

You  will  be  one  of  100  total  subjects  participating  in  this  study  being  conducted  by  Dr.  Colin 
Mackenzie  and  colleagues  at  Anatomy  Board,  University  of  Maryland  School  of  Medicine, 
Baltimore,  in  collaboration  with  the  Uniform  Services  University  of  Health  Sciences  (USUHS) 
Bethesda,  Maryland 

PROCEDURES 

Stud)’  staff  will  provide  initial  information  about  the  stud)'  by  e-mail,  including  copies  of  the 
consent  form  and  descriptions  of  the  data  to  be  collected  on  each  participant  If  you  express 
interest  in  participating  m  the  study  you  will  be  provided  with  likely  dates  the  ASSET  Stud)'  will 
be  conducted.  On  the  day  of  ASSET  Stud)'  participation,  time  will  be  set  aside  to  discuss  the 
stud)'  and  the  consent  process  during  which  all  procedures,  and  alternatives,  risks  and  benefits, 
associated  with  participation  in  ASSET  Study  will  be  discussed.  You  will  be  encouraged  to  ask 
questions  at  any  time  during  this  process  and  throughout  the  stud)'. 


Page  1  of  5 


HP-00054443  UM  IRB  Approval  Date  11/17/2014 
Do  Not  Sign  this  Form  after  this  Date  11/16/2015 


85 


Appendix  44:  Kick-Off  Meeting  Agenda  February  14th,  2013 


Kick-Off  Meeting  Agenda  February  14* 

ASSET  Funding  Kick-Off  Meeting 

Thursday,  February  14, 2013, 12:30pm  -  4:00pm;  Executive  Board  Room  HSF II 


Final  Agenda 

12:30 

Check  in  and  light  Lunch 

1:00-  1:15  pm 

Introduction 

Brnce  Jarrell  Chief  Academic  and  Research  Officer  (CARO), 

Senior 

Vice  President,  and  Dean  of  the  Graduate  School  UM  Baltimore 
Tom  Scalea,  Professor  of  Surgery,  Director  Shock  Trauma  Center 

1:15 -1:30  pm 

Medical  Simulation 

Mr  Tony  Story'  (via  Teleconference) ,  Telemedicine  &  Advanced 
Technology  Research  Center  (TATRC).  Armed  Forces  Simulation  Institute  for  Medicine 


1:30 -1:45  pm 

ASSET  Overview  and  Summary'  Statistics 

Col  (Rtd)  Mark  Bowyer,  MD  FACS,  Director  of  Surgical 

Simulation 

The  Normal  M  Rich  Dept  of  Surgery'  Uniformed  Services 

University'  (USUHS) 


1:45 -2:00  pm 

ASSET  History'  at  UMB 

Sharon  Henry'.  MD  FACS,  UMB 

2:00  -2:15  pm 

USAF  Military  Perspective  on  ASSET  and  Study- 

Col  Stacy  Shackelford,  MD  FACS,  Director  C-STARS  Baltimore 

2:15 -2:45  pm 

Study  overview  and  SOW 

Colin  Mackenzie.  PL  UMB  /  STAR  ORC 

2:45  -  3:00  pm 

Cognitive  Task  Analysis(via  Teleconference) 

Valerie  Shalin,  PhD.  Wright  State  University 

3:00  -  3:20 pm 

Break 

3:20-3:40  pm 

Maryland  State  Anatomy  Board 
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Appendix  45:  RCI  Invoice  for  AV  Hardware 


Appendix  2:  RCI  Invoice  for  AV  Hardware 
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Appendix  46:  UMB  Invoice  for  TRR  System 


Appendix  3:  UMB  invoice  for  TRR  system 


Tax  Invoice 

AlfredHealth 

ABN:  27  318  956  319 

Alfred  Health  incorporates  The  Alfred.  CautfieW  Hospital  and  Sandringham  Hospital 
Finance  Department,  The  Alfred,  PO  Box  315.  Prahran  VIC  3181 
Telephone  03  9076  5442  Fax:  03  9076  2102 


UMB  Aococnts  Payable  Department 
Saratoga  Street  Offices 
220  Arch  Street 
Rm.  02-123 
Baltimore  MD  21201 

UNITED  STATES  OF  AMERICA  email:  accounts. recervable@alfred  org  au 


Description  Qty  Rate  Pre  GST  Amt  GST  Amt  Ful  Amount 


sundry  income  1  273.405230  273.405230  0  00  273.40523 

Purchase  Order  PUR0 1-0000024662 

3lte  &  3ubject  Spec  TV;  License  TRR  3oto*ore  ncUSng  TRR  Algorithm 

Designer  Tool  scripted  tor  A33ET 

Redesigned  User  Interface  tor  TRR/A33ETT 

TRR  \gortfr  Design  Trafritog  Support 

TRR  Aflorten-  Testing 

3lmua0on  framing  and  vcrtftaBor 

Documentation  Version  Confroi  &  user  Manuals 

3ysten-s  montorfrig  and  technics  support  for  3  years 
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Total 


000  273,40523 


Payment  Received  0.00 

Total  Payable  273.40523 


To  ensure  the  correct  rtentificatjon.  please  detach  the  slip  below  and  return  it  with  your  payment. 


Payment  Terms:  14  Days 

X - 


Payment  From 


UMB  Accounts  Payable  Department 
UMB002 


Bank  Card  Master  Card  VISA  Diners  Amex 


EFT  Payment  Details 

Please  ensure  that  Invoice  number  Is  quoted  and  remittance  advice  is  faxed  or  emabed  to  the  above  address 
Weetpoc  BSB:  0SHJ79  Account  Mo:  114772 


Name  on  card: 


Type 
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Appendix  47:  Assessing  Surgical  Training:  a  Utility  Analysis  of  the  Advanced  Surgical  Skills  for 
Exposure  in  Trauma  Course 


The  Norman  M  Rich  Department  of  Surgery.  Uniformed  Services  University  of  Health 
Sciences 


Abstract 

Background:  Surgical  experience  with  managing  traumatic  hemorrhage  has  declined  m 
training  programs  and  in  practice  To  address  this,  the  American  College  of  Surgeons 
launched  the  Advanced  Surgical  Skills  for  Exposure  m  Trauma  (ASSET)  course  m  2010, 
a  human  cadaver-based  course  to  review  the  anatomy,  skills  and  techniques  for  rapid 
vascular  exposures. 

Study  design:  We  compared  self-reported  confidence  of  participants  (n=523)  with 
surgical  tasks  (n=47)  at  baseline  and  directly  after  ASSET  training  to  examine  the  effect 
of  training.  Median  pre-  and  post-training  self-reported  confidence  scores  were  assessed 
by  Wilcoxon  matched  pairs  test  directional  change  by  Freeman-Halton  contingency  tests, 
and  relative  improvement  for  specific  procedures  using  utility'  values  assigned  for  each 
possible  combination  of  pre-  and  post-training  confidence  levels. 

Results:  All  surgeons  recorded  improved  confidence  in  all  five  anatomic  body  regions 
after  ASSET  training  (p<0.0001).  Following  the  course,  surgeons  reported  a  high 
confidence  level  in  78%  of  the  47  procedures.  The  body  region  most  improved  by 
ASSET  training  was  the  upper  limb,  with  49%  of  surgeons  improving  from  low  to  high 
confidence  (Freeman-Halton  1x3  p=0.017).  Residents/fellows  achieved  the  greatest 
improvement  m  confidence  levels  The  highest  utility  value  occurred  with  pelvic 

-J- 

Distribution  A;  Approved  far  public  release;  distribution  is  unlimited.  Case  Number;  SBAFW -2014- on ,  a  Feb  2014 
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Appendix  48:  OEI  Invoice  for  Physical  Model  Delivery 


Appendix  5:  OEI  invoice  for  Physical  models 
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Appendix  49:  Evaluation  Script  Slides  and  Video  Evaluation  Sheet 


Evaluation  Sheet  Examples 


AXILLARY  ARTERY  EXPOSURE  GLOBAL  RATING  (cirde  one): 


Overall  understanding  of  the  Evaluation  and  Treatment  of  a  Patient  with  a  Suspected  Axillary 

I 

1 

1 

2 

3 

4 

5 

UTA* 

Core  knowledge  is  poor 
and  there  is  no 

evidence  of 
understanding  the 
nuances  of  evaluation 

and  diagnosis. 

Core  knowledge  is  faw 
with  some 

understanding  of  the 
nuances  of  evaluation 
and  diagnosis. 

Core  knowledge  is  good 
with  moderate 
understanding  of  the 
nuances  of  evaluation 
and  diagnosis- 

Core  knowledge  is  very 
good  with  thorough 
understanding  of  the 
nuances  of  evaluation 
and  diagnosis. 

Core  knowledge  is 
excellent  with  a 
superior  understanding 
of  the  nuances  of 

evaluation  and 

diagnosis. 

Overall  Understanding  of  the  Surgical  Anatomy  of  the  Axillary  Region: 

1 

2 

3 

4 

S 

UTA* 

Poor  knowledge  of 
the  regional  anatomy. 
Unable  to  identify 
major  structures  or 
their  relationships. 

Fair  knowledge  of 
regional  anatomy.  Can 
name  some  of  the  major 
structures  and  their 
relationships 

Good  understanthng  of 
the  anatomy.  Can  name 
most  of  the  major 
structures  and  their 
relationships. 

Very  good 
understanding  of 
anatomy.  Able  to  point 
out  all  of  the  major 
structures  and  their 
relationships. 

Excellent 

understanding  of  the 
anatomy,  including 
variants.  Knows  the 

minutia.  Should  be 
teaching  anatomy 
class. 

Technical  Skits  for  Exposing  Axitory  Artery: 


1 

2 

3 

4 

5 

UTA* 

The  participant's 

The  participant 

The  participant 

The  participant 

The  participant 

technical  skills  were 

demonstrated  fair 

demonstrated  good 

demonstrated  very 

demonstrated 

poor  with  much  wasted 

technical  skits  with 

technical  skills  with 

good  technical  skits 

excellent  technical 

moves  and  very  poor 

some  wasted 

occasional  wasted 

with  minimal  wasted 

skills  with  no  wasted 

tissue  handing. 

movements  and  errors 
in  tissue  handling 

movements  and  errors 
in  tissue  handing. 

movements  and  errors 
in  txsaje  handling. 

movements  and 
proper  respect  for 
tissues. 

This  participant  is  ready  to  perform  exposure  and  control  the  Axillary  Artery: 


1 

2 

3 

4 

5 

UTA* 

Take  me  to  another 
hospital  please! 

This  participant  could 
do  the  exposure  fine 
with  experienced  help, 
but  will  struggle  if  left 
alone. 

The  participant  might 
need  to  look  at  a  text  to 
refresh  their  memory 
but  will  be  able  to 
perform  the  exposure. 

This  individual  wil  be 
able  to  perform  the 
exposure  with  minimal 
difficulty  in  an 
expeditious  fashion. 

Absolutely.  1  hope  that 
this  individual  is  on  call 
if  1  am  injured. 

overall  rating  (1-100):  Body  Habitus  o(  cadaver  [circle!  cadaver  anatomy  .'circle!: 

Obese  Average  Thin  Normal  Variant 

a  90  Excellent  I  hope  that  this  individual  is  on  call  if  I  am  injured 

80-89  This  indnhdual  will  be  able  to  perform  the  exposure  with  minimal  drffi cutty  in  an  expeditious  fashion 
70-79  The  participant  might  need  to  look  at  a  text  to  refresh  their  memory  but  will  be  able  to  perform  the 
exposure 

60-69  This  participant  could  do  the  exposure  with  experienced  help,  but  will  struggle  rf  left  alone 
<60  Take  me  to  another  hospital  please! 


•UTA  (Unable  to  Assess):  The  detail  for  this  determination  was  not  possible  from  the  video 
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Appendix  50:  Development  of  a  Surgical  Skills  Assessment  Method  for  Trauma 

Stacy  Shackelford,  MD,  FACS,  Evan  Garofalo,  PhD,  Megan  Holmes,  BS,  Hegang  Chen  PhD, 
Mark  Bowyer,  MD,  FACS  ,  Sharon  Henry,  MD,  FACS,  Babak  Sarani,  MD,  FACS,  Jason  Pasley, 
MD,  Colin  Mackenzie,  MBChB 

Background:  With  limits  on  residency  training  hours  and  decrease  in  penetrating  trauma 
nationally,  surgical  experience  with  managing  traumatic  hemorrhage  has  declined.  An  objective 
assessment  of  surgical  skills  in  trauma  would  be  useful  in  many  training  situations,  to  include 
course  development,  residency  training,  board  certification  and  preparation  for  military 
deployment.  We  hypothesized  that  performance  metrics  for  trauma  surgery  can  reliably 
distinguish  expert  from  novice  surgeons. 

Study  Design:  We  performed  a  video  task-analysis  of  10  attending  trauma  surgeons  and  10 
general  surgery  residents  during  performance  of  three  vascular  exposures  (axillary,  brachial, 
femoral  arteries)  and  lower  extremity  fasciotomy.  Performance  characteristics  of  expert  and 
novice  surgeons  were  identified  and  used  to  develop  a  technical  skills  metric  score.  The  score 
includes  completion  of  specific  surgical  steps  and  assessment  of  surgical  technique.  Five 
evaluators  scored  blinded  videos  of  the  four  procedures.  Interrater  reliability  was  assessed  using 
intraclass  correlation  coefficient  (ICC).  Expert  and  novice  scores  were  compared  using  Kruskal- 
Walis  test. 

Results:  Discriminating  characteristics  with  best  evaluator  ICC  between  expert  and  novice 
technical  skills  included  obtains  necessary  exposure  (p<0.00001),  performing  procedures  without 
unnecessary  dissection  (p<0. 00001),  proceeds  at  appropriate  pace  (p<0. 00001),  and  performs 
procedure  with  a  logical  sequence  (p=0. 00001).  ICC  displayed  in  table. 

Conclusion:  A  surgical  technical  skills  metric  score  can  discriminate  expert  from  novice 
performance  required  to  complete  four  surgical  procedures  through  the  use  of  discriminating 
performance  characteristics  that  may  be  useful  for  objective  surgical  skill  assessment. 

Intraclass  Correlation  Coefficient 


Axillary 

artery 

Technical  Skill 

exposure 

Obtains  necessary  exposure 

0.98 

No  unnecessary  dissection 

0.96 

Proceeds  at  appropriate  pace 

0.97 

Brachial 

Femoral 

artery 

artery 

Fasciotomy 

exposure 

exposure 

0.92 

0.79 

0.97 

0.91 

0.96 

0.94 

0.88 

0.94 

0.97 
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Performs  with  logical  sequence 


0.93 


0.87 


0.97 


0.95 


Appendix  51:  Video  of  an  Expert  Performing  the  Axillary  Artery  Exposure 


a. 


Appendix  51  -  Expert  Axillary  Artery  Exposure.wmv 
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Appendix  52:  Updated  Script  and  Powerpoint  Slides  Used  for  Evaluations 


Date: 

(Circle  timing):  Pre  Post 

1st  Trial 

Circle  type  of  trial:  Cadaver  /  Model 


Name  of  Evaluator: 
Name  of  Candidate: 


Case  One:  Axillary  Artery 

Case  Presentation: 

•  You  are  called  to  the  Emergency  Department  to  see  a  24  y/o  male  who 
was  shot  during  an  attempted  robbery  sustaining  a  single  gunshot  wound 
to  the  upper  anterior  lateral  Right/Left  Chest. 

•  He  was  reported  to  have  a  large  amount  of  bright  red  blood  at  the  scene, 
but  is  currently  not  bleeding. 

•  He  is  complaining  of  pain  at  the  site  of  the  wound  and  inability  to  move 
his  arm. 


[Advance  slide  to  show  image  of  wound] 

[Advance  slide  to  continue  narrative] 

•  He  is  awake  and  talking  with  bilateral  and  equal  breath  sounds  and  a  BP 
of  80/60  and  a  heart  rate  of  130  after  2  liters  of  lactated  ringers 

•  There  is  a  single  wound  as  seen  with  no  other  obvious  trauma  and  no 
"exit  wound".  His  hand  is  cool  and  pale. 


l 


Advanced  Surgical  Skills 
for  Exposure 
in  Trauma  Course 


American  College  of  Surgeons 
Committee  on  Trauma 


94 


Appendix  53:  Critical  Errors  in  Rarely  Performed  Procedures  0.5-5  Years  After  Training  Among 
85  Surgeons 


Colin  F  Mackenzie,  Kristy  Pugh,  Guinevere  Granite,  Hegang  Chen,  Adam  Puche,  Samuel 
Tisherman.  Shock  Trauma  Anesthesiology  Research,  Departments  of  Epidemiology,  Anatomy 
and  Neurobiology,  Surgery,  University  of  Maryland  School  of  Medicine,  Baltimore 

Background:  Technical  failures  in  individual  clinical  skills  occur  during  surgery  and  are  known 
to  increase  post-operative  morbidity  and  mortality  (1).  Graduating  general  surgery  residents  have 
little  experience  with  many  procedures  needed  for  trauma,  e.g.,  brachial  artery  exposures  or 
lower  extremity  fasciotomy  (2).  The  Advanced  Surgical  Skills  for  Exposure  in  Trauma  (ASSET) 
course  was  developed  to  correct  this  deficit.  We  hypothesized  that  the  occurrence  of  critical 
technical  errors  (e.g.,  vessel  loop  encircled  wrong  structure)  and  critical  management  errors  (e.g., 
life-threatening  delays)  decrease  with  training  and  subsequently  increase  with  post-training 
interval  skill  decay.  Skill  decay  in  complex  procedures  occurs  widely,  including  for 
anesthesiologist’s  procedural  skills  (3). 

Methods:  Surgeons  0.5-5  years  after  ASSET  training  were  video  recorded  performing  axillary, 
brachial,  and  femoral  artery  exposure  and  control  (encircle  with  double  vessel  loop),  and  a  4 
compartment  lower  extremity  fasciotomy  (FAS)  on  unpreserved  cadavers.  Skills  were  evaluated 
by  two  trained,  co-located  evaluators  with  a  standardized  script  and  a  validated  individual 
procedure  score  (IPS)  metric  (4).  Linear  mixed  modelling  included:  anatomy  skills,  years  and 
operative  experience  since  training,  and  cadaver  body  habitus. 

Results:  All  4  procedures  were  performed  by  85  surgeons,  Group  2:  forty  2nd  -  6th  year  residents 
before  and  after  training  and  1-1.5  years  later,  Group  3:  35  practicing  surgeons  2.5  years  after 
training,  and  10  Experts  (practicing  [mean  16  years]  as  trauma  attending  surgeons). For  vascular 
procedures,  among  Group  2  surgeons,  60%  critical  error  rate  decreased  to  19%  (P<  0.001) 
immediately  after  training,  a  rate  comparable  to  experts  (15%).  There  was  no  difference  in  error 
rates  post-training  out  to  1-1.5  years  (22%).  However,  Group  3  surgeons  error  rates  increased  to 
36,5%  (P<0.003)  and  error  recovery  decreased  compared  to  all  other  surgeons  after  training 
(Figure).  A  similar  pattern  was  observed  for  FAS.  Only  10%  of  Group  2  surgeons  decompressed 
all  four  compartments  before  ASSET,  which  improved  (p<  0.02)  to  50%  post-training 
comparable  to  60%  4  compartment  decompression  among  experts).  However,  only  35%  of 
Group  3  surgeons  decompressed  all  4  compartments,  fewer  (p  <0.03)  than  all  other  surgeons 
after  training.  Among  Group  3  surgeons,  error  recovery  was  lowest  (p<0.05)  for  the  3  vascular 
procedures  and  less  FAS  compartments  were  decompressed  mean  2.5  years  after  training  (p  < 
0.03).  Four  experts  failed  decompress  =/>  1  compartment.  Critical  errors  correlated  with  IPS 
measured  lack  of  correct  anatomic  landmarks  and  procedural  steps,  suggesting  mitigation  efforts 
may  be  amenable  to  focussed  training  interventions  (5). 

Conclusion:  Occurrence  of  critical  errors  improved  with  training,  but  skills  decay  was  detectable 
mean  2.5  years  later.  Refresher  strategies,  concentrating  on  anatomy,  are  required  to  minimize 
skills  decay  in  rarely  used  procedural  skills,  even  among  experts. 
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Critical  Technical  Errors  &  Error  Recovery  Pre  and 
Post-ASSET  and  mean  1.2  and  2.5  Years  Later 


ASSET 

Training 

Mean  1.2 

years 

al 

Fter  A 

1 

1 

u 

J 

Mean  2.5 
years 


afterASSET  B  Critical  Technical  Errors 
occurred 

■  %  Error  Recovery 


Are-ASS  ET 
Phase  2 


Post^ASSET  detention  Phase  Phase  3 
Phase  2  2 

Study  Arm 
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Supporting  Data 

Table  1:  Median  Confidence  with  Surgical  Anatomy  (Phase  II) 


Median  reported  levels  of  confidence  with  the  surgical  anatomy  and  comfort  to  performance  surgical  procedures 
independently  before  and  after  ASSET  training  (n=23) 


Pre-Training  Post-Training 


Pre- 

Post- 

Pre- 

Post- 

evaluation 

evaluation 

evaluation 

evaluation 

Shoulder/ axillary  region 

2.5 

2 

4 

4 

Arm 

3 

3 

3.5 

4 

Understanding  of  the 
surgical  anatomy: 

Forearm 

Ingunal  region 

3 

4 

3 

3 

4 

4 

4 

4 

Lower  extremity 

3 

4 

4 

4 

Shoulder  region  for  traumatic  injury 

2 

2 

3 

4 

Performing  surgical 

Arm  for  traumatic  injury 

2 

3 

4 

4 

procedures  for  traumatic 

Forearm  fortraumatic  injury 

2 

3 

4 

4 

injury  independently: 

Inguinal  region  for  traumatic  injury 

3 

4 

4 

4 

Lower  extremity  fasciotomy  fortraumatic  injury 

3 

3 

4 

4 

These  data  indicate  that  surgeons  have  moderate  confidence  in  their  understanding  of  the 
relevant  anatomy  before  beginning  ever  performing  the  procedures.  Their  confidence  increases 
after  the  initial  procedure  performance  and  continues  to  increase  following  their  post-training 
evaluation. 

Phase  2  surgeons  have  low  to  moderate  confidence  initially  in  their  ability  to  perform  procedures 
independently  but  they  gain  higher  levels  of  confidence  after  deliberate  practice  performing  the 
procedures. 
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Figure  1:  Mean  Trauma  Readiness  Index  by  Experimental  Trial 


Trial 

These  data  indicate  a  significant  improvement  after  training,  a  lower  score  for  surgeons  2  to  5 
years  after  training  and  a  significantly  higher  score  for  Expert  surgeons. 

Table  2:  Pair  Wise  Comparisons  of  TRI  between  Surgeon  Types 

(Tukey  unequal  n,  a=0.05,  red  indicates  significant  comparisons) 


Pair-wiseTrial 


Trial 

Mean  TRI 

Pre-training 

Post-training 

Retention 

Pre-training 

0.46 

Post-training 

0.61 

0.000008 

Retention 

0.57 

0.00004 

0.2 

Expert 

0.68 

0.000008 

0.02 

0.00004 
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Figure  2:  Score  (%)  for  Individual  Components  of  TRI  before  and  after  Training 


(box  plot:  mean,  1  SE,  and  whiskers=2  SD) 


These  data  indicate  that  the  greatest  improvement  after  training  is  seen  for  anatomical 
knowledge,  technical  skills,  and  procedural  steps. 
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Figure  3:  Successful  Compartment  Decompressions  among  Surgeon  Types 


Procedure  Time  (min) 

^..Pre-training 
'^.Post-training 
A  Retention 
*  Expert 


This  figure  indicates  the  number  of  successfully  decompressed  compartments  during  a  2  incision 
4  compartment  lower  extremity  fasciotomy  by  time  performed  by  surgeons  in  each  experimental 
group.  Fit  lines  with  95%  Cl  are  shown  for  Pre  and  Post-training  residents  only.  Post-training 
surgeons  and  experts  tend  to  decompress  more  compartments  in  about  the  same  amount  of  time 
as  Pre-trained  surgeons  decompress  fewer  compartments  (fail  to  decompress  the  compartments). 
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Individual  Procedure  Score 


Figure  4:  Individual  Procedure  Score  (IPS)  and  TRI  for  all  Four  Procedures 


(box  plot:  mean,  1  SE,  and  whiskers=l  SD) 
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Table  3  A  :  Completed  SF  425  form 


FEDERAL  FINANCIAL  REPORT 

1.  Federal  Agency  and  Organliational 

Element  to  Which  Report  Is  Submitted 

US  Army  Medical  Research 

2.  Federal  Grant  or  Other  Identifying  Number  Assigned  by  Federal  Agency  Page  of 

(To  report  multiple  grants,  use  FFR  Attachment  1  1 

W81XWH-13-2-0028  pages 

3.  Recipient  Organization  (Name  and  comple 

University  of  Maryland,  Baltimore,  Sp 

te  address,  including  ZIP  code) 

onsored  Projec  ts  Accounting  and  Complianc  e,  220  Arch  Street,  Baltimore  MD,  21201 

4a.  DUNS  number 


188435911 


52-6002033 


5.  Recipient  Account  Number  or  Identifying  Number 


10011190 


Quarterly 

Annual 


l=B: 


i-Annual 

nal 


a 

ra 


7.  Basis  of  Accounting 


Cash 


Accrual 


□ 


.  Project/Grant  Period 
From:  (Month,  Day,  Year) 

02/15/13 


To:  (Month,  Day,  Year) 

02/14/16 


3.  Reporting  Period  End  Date 
(Month,  Day,  Year) 


02/14/16 


10.  Transactions 


(Use  lin  a-c  for  single  or  multiple  grant  reporting) 


Federal  Cash  (To  report  multiple  grants,  also  use  the  FFR  Attachment) 


a.  Cash  Receipts 


1,862,757.25 


b.  Cash  Disbursements 


2,068,387.82 


c.  Cash  on  Hand  (line  a  minus  b) 


(205,630.57) 


(Use  lines  d-o  for  single  grant  reporting) 


Federal  Expenditures  and  Unobligated  Balance: 


d.  Total  Federal  funds  authorized 


2,068,387.82 


e.  Federal  share  of  expenditures 


2,068,387.82 


f.  Federal  share  of  unliquidated  obligations 


g.  Total  Federal  share  (sum  of  lines  e  and  f) 


2,068,387.82 


h.  Unobligated  Balance  of  Federal  funds 


Recipient  Share: 


I,  Total  recipient  share  required 


j.  Recipient  share  of  expenditures 


k.  Remaining  recipient  share  to  be  provided  (line  i  minus  j) 


0.00 


Program  Income: 


I.  Total  Federal  program  income  earned 


m.  Program  income  expended  in  accordance  with  the  deduction  alternative 


n.  Program  income  expended  in  accordance  with  the  addition  alternative 


0.00 


o.  Unexpended  program  income  (line  I  minus  line  m  or  line  n) 


0.00 


11.  Indirect 

a.  Type 

b.  Rate 

c.  Period  From  To 

d.  Base 

e.  Amount  charged 

f.  Federal  Share 

Expense 

Pre-Determined 

26.00% 

02/15/13  -  02/14/16 

1,528,293.78 

397,356.40 

397,356.40 

Pre-Determined 

0.00% 

0.00 

0.00 

0.00 

12.  Remarks:  Attach  any  explanations  deemed  necessary  or  information  required  by  Federal  sponsoring  agency  in  compliance  with  the  governing  legislation 

Prepared  by:  Andrew  Rice_ Email: andrcw.ricc@uman  land.edu 


13. Certification:  By  signing  this  report,  I  certify  to  the  best  of  my  knowledge  and  belief  that  the  report  is  true,  complete  and  accurate,  and  the  expenditures, 

disbursements  and  cash  receipts  are  for  the  purposes  and  intent  set  forth  in  the  award  documents.  I  am  aware  that  any  false,  fictitious, 
or  fraudulent  information  may  subject  me  to  criminal,  civil  or  administrative  penelaties.  (U.S>  Code,  Title  18,  Section  1001) 


Typed  or  Printed  Name  and  Title 


Krista  Salsberg,  Manager 
Sponsored  Projects  Accounting  and  Compliance 


Telephone  (Area  code,  number  and  extension  410-706-6786 


Email  address  ksalsberg@umaryland.edu 


Signature  of  Authorited  Certifying  Official 


Date  Report  submitted 


j/wusisfa-- 


04/07/2016 


Standard  Form  425  OMB  Approval  Number;  0348-0061  Expiration  Date;  10/31/2011 
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Table  3  B:  Expenditures  for  the  Quarter  Ending  02/15/16 


NEW  QUARTER  (12/01/2014  -  02/28/2015) 


Cost  Elements 

Quarter  Total 

Personnel 

$93,198.37 

Fringe  Benefits 

$19,085.25 

Supplies 

$776.00 

Equipment 

-$11,400.00 

Travel 

$3,238.84 

Other  Direct  Costs 

$37,089.89 

Subtotal 

$141,988.35 

Indirect  Costs 

$39,880.96 

Fee 

$0.00 

Total  Expenditures 

$181,869.31 
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Table  4:  Current  Personnel  Effort 


Personnel 

Role 

Percent  Effort 

Mackenzie,  Colin 

Principal  Investigator 

60% 

Hu,  Peter 

Co-Investigator 

5% 

Hagegeorge,  George 

Senior  Technician 

60% 

Chen,  Hegang 

Statistician 

21% 

Granite,  Guinevere 

Research  Coordinator 

60% 

Pugh,  Kristy 

Research  Assistant 

100% 

Teeter,  William 

Surgical  Resident 

25% 

Stansbury,  Lynn 

Research  Assistant 

8% 

Yang,  Shiming 

Research  Associate 

35% 
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Figure  5:  Prediction  Fit  Plots:  a)  BA  adj  IPS  using  AA  adj  IPS;  b)  BA  reg  IPS  using  FA  and  A  A  adj 
IPS;  c)  FA  reg  IPS  using  AA  reg  IPS 


Contour  Fit  Plot  for  BA_adjusted_IPS_mean 


Figure  5a.  Predicting  Brachial  Artery  (BA)  adjusted  Individual  Procedure  Score  (IPS) 
using  Axillary  Artery  (AA)  and  Femoral  Artery  (FA)  adjusted  IPS 


Contour  Fit  Plot  for  BA_regular_IPS_mean 


Figure  5b.  Predicting  BA  regular  IPS  using  AA  and  FA  regular  IPS 


Fit  Plot  for  FA_regular_IPS_mean 


Figure  5c.  Predicting  FA  regular  IPS  using  AA  regular  IPS 
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Figure  6:  Regular  IPS  and  TRI  Retention  Plots 


Monday,  January  04,  2016  5:31  PM  1 
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Page  1 :  Regular  Trauma  Readiness  Index  (TRI)  scores  and  regression  lines  for  all  4  procedures  (Axillary 
Artery,  Brachial  Artery,  Femoral  Artery,  and  Lower  Leg  Fasciotomy)  for  Phase  2  participants  after  the 
ASSET  course  (2_post)  and  12  to  18  months  after  ASSET  (2_18mfol),  Phase  3  participants  2  to  5  years 
after  ASSET  (3_retent),  and  Experts  (2_expert),  and  how  they  correlate  with  50,  75,  and  90%  expert 
tertiles.  This  longitudinal  graph  demonstrates  the  performance  retention  of  all  surgeons  evaluated  on  the  4 
procedures  at  specific  time  intervals  since  taking  the  ASSET  course. 

Page  2:  Regular  Trauma  Readiness  Index  (TRI)  scores  and  regression  lines  for  all  4  procedures  (Axillary 
Artery,  Brachial  Artery,  Femoral  Artery,  and  Lower  Leg  Fasciotomy)  of  the  Phase  2  participants  12  to  18 
months  after  ASSET,  Phase  3  participants  2  to  5  years  after  ASSET,  and  Experts  (labeled  as  all),  and  how 
they  correlate  with  the  50,  75,  and  90%  expert  tertiles.  This  graph  demonstrates  surgical  performance 
retention  for  these  3  participant  groups  evaluated  longitudinally  on  all  4  procedures. 

Page  3:  Regular  Trauma  Readiness  Index  (TRI)  scores  and  regression  lines  for  all  vascular  procedures 
(Axillary  Artery,  Brachial  Artery,  and  Femoral  Artery)  for  Phase  2  participants  after  the  ASSET  course 
(2_post)  and  12  to  18  months  after  ASSET  (2_18mfol),  Phase  3  participants  2  to  5  years  after  ASSET 
(3_retent),  and  Experts  (2_expert),  and  how  they  correlate  with  50,  75,  and  90%  expert  tertiles.  This 
longitudinal  graph  demonstrates  the  performance  retention  of  all  surgeons  evaluated  on  the  3  vascular 
procedures  at  specific  time  intervals  since  taking  the  ASSET  course. 

Page  4:  Regular  Trauma  Readiness  Index  (TRI)  scores  and  regression  lines  for  all  vascular  procedures 
(Axillary  Artery,  Brachial  Artery,  and  Femoral  Artery)  of  the  Phase  2  participants  12  to  18  months  after 
ASSET,  Phase  3  participants  2  to  5  years  after  ASSET,  and  Experts  (labeled  as  all),  and  how  they 
correlate  with  the  50,  75,  and  90%  expert  tertiles.  This  graph  demonstrates  surgical  performance  retention 
for  these  3  participant  groups  evaluated  longitudinally  on  the  3  vascular  procedures. 

Page  5:  Regular  Axillary  Artery  (AA)  procedure  mean  Individual  Procedure  Score  (IPS)  scores  and 
regression  lines  for  Phase  2  participants  after  the  ASSET  course  (2_post)  and  12  to  18  months  after 
ASSET  (2_18mfol),  Phase  3  participants  2  to  5  years  after  ASSET  (3_retent),  and  Experts  (2_expert),  and 
how  they  correlate  with  50,  75,  and  90%  expert  tertiles.  This  longitudinal  graph  demonstrates  the  AA 
procedure  performance  retention  of  all  surgeons  evaluated  at  specific  time  intervals  since  taking  the 
ASSET  course. 

Page  6:  Regular  Axillary  Artery  (AA)  procedure  mean  Individual  Procedure  Score  (IPS)  scores  and 
regression  lines  for  the  Phase  2  participants  12  to  18  months  after  ASSET,  Phase  3  participants  2  to  5 
years  after  ASSET,  and  Experts  (labeled  as  all),  and  how  they  correlate  with  the  50,  75,  and  90%  expert 
tertiles.  This  graph  demonstrates  surgical  performance  retention  for  these  3  participant  groups  evaluated 
longitudinally  on  the  AA  procedure. 

Page  7 :  Regular  Brachial  Artery  (BA)  procedure  mean  Individual  Procedure  Score  (IPS)  scores  and 
regression  lines  for  Phase  2  participants  after  the  ASSET  course  (2_post)  and  12  to  18  months  after 
ASSET  (2_18mfol),  Phase  3  participants  2  to  5  years  after  ASSET  (3_retent),  and  Experts  (2_expert),  and 
how  they  correlate  with  50,  75,  and  90%  expert  tertiles.  This  longitudinal  graph  demonstrates  the  BA 
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procedure  performance  retention  of  all  surgeons  evaluated  at  specific  time  intervals  since  taking  the 
ASSET  course. 

Page  8:  Regular  Brachial  Artery  (BA)  procedure  mean  Individual  Procedure  Score  (IPS)  scores  and 
regression  lines  for  the  Phase  2  participants  12  to  18  months  after  ASSET,  Phase  3  participants  2  to  5 
years  after  ASSET,  and  Experts  (labeled  as  all),  and  how  they  correlate  with  the  50,  75,  and  90%  expert 
tertiles.  This  graph  demonstrates  surgical  performance  retention  for  these  3  participant  groups  evaluated 
longitudinally  on  the  BA  procedure. 

Page  9:  Regular  Femoral  Artery  (FA)  procedure  mean  Individual  Procedure  Score  (IPS)  scores  and 
regression  lines  for  Phase  2  participants  after  the  ASSET  course  (2_post)  and  12  to  18  months  after 
ASSET  (2_18mfol),  Phase  3  participants  2  to  5  years  after  ASSET  (3_retent),  and  Experts  (2_expert),  and 
how  they  correlate  with  50,  75,  and  90%  expert  tertiles.  This  longitudinal  graph  demonstrates  the  FA 
procedure  performance  retention  of  all  surgeons  evaluated  at  specific  time  intervals  since  taking  the 
ASSET  course. 

Page  10:  Regular  Femoral  Artery  (FA)  procedure  mean  Individual  Procedure  Score  (IPS)  scores  and 
regression  lines  for  the  Phase  2  participants  12  to  18  months  after  ASSET,  Phase  3  participants  2  to  5 
years  after  ASSET,  and  Experts  (labeled  as  all),  and  how  they  correlate  with  the  50,  75,  and  90%  expert 
tertiles.  This  graph  demonstrates  surgical  performance  retention  for  these  3  participant  groups  evaluated 
longitudinally  on  the  FA  procedure. 

Page  11:  Regular  Lower  Leg  Fasciotomy  (FAS)  procedure  mean  Individual  Procedure  Score  (IPS)  scores 
and  regression  lines  for  Phase  2  participants  after  the  ASSET  course  (2_post)  and  12  to  18  months  after 
ASSET  (2_18mfol),  Phase  3  participants  2  to  5  years  after  ASSET  (3_retent),  and  Experts  (2_expert),  and 
how  they  correlate  with  50,  75,  and  90%  expert  tertiles.  This  longitudinal  graph  demonstrates  the  FAS 
procedure  performance  retention  of  all  surgeons  evaluated  at  specific  time  intervals  since  taking  the 
ASSET  course. 

Page  12:  Regular  Lower  Leg  Fasciotomy  (FAS)  procedure  mean  Individual  Procedure  Score  (IPS)  scores 
and  regression  lines  for  the  Phase  2  participants  12  to  18  months  after  ASSET,  Phase  3  participants  2  to  5 
years  after  ASSET,  and  Experts  (labeled  as  all),  and  how  they  correlate  with  the  50,  75,  and  90%  expert 
tertiles.  This  graph  demonstrates  surgical  performance  retention  for  these  3  participant  groups  evaluated 
longitudinally  on  the  FAS  procedure. 
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Figure  7  (a-d):  IPS  Longitudinal  Data  Graphs 


Figure  7a.  Longitudinal  Data  Graphs  of  Phase  2  Pre,  Post,  and  Follow-up  Individual  Procedure  Score  (IPS  or  procedure 
score  %)  scores  for  4  of  the  5  evaluated  procedures  (Axillary  Artery  (AA),  Brachial  Artery  (BA),  Femoral  Artery  (FA), 
and  Lower  Leg  Fasciotomy  (FAS)),  and  Phase  2  follow-up  IPS  scores  for  Carotid  Artery  (CA),  and  how  these  IPS  scores 
correlate  to  the  3  tertile  rankings  (lowest,  middle,  and  top)  for  each  evaluation  time  interval. 
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Figure  7b. 
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Figure  8:  Benefits  of  ASSET  Training  on  Surgical  Skills 
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Figure  8.  Surgical  Skills  Score  Changes  Post-ASSET:  Procedural  Steps  Skills  Increased  57%;  Anatomic  Skills  (i.e. 
Correct  Landmarks,  Skin  Incision,  etc.)  Increased  43%;  Technical  Skills  Increased  25%;  Vascular  Procedures  Trauma 
Readiness  Index  Increased  28%;  1.9  to  3  minutes  Decrease  (mean  2.5  minutes)  in  Vascular  Procedure  Time 


110 


Figure  9  (a  &  b):  Individual  Procedure  Scores  (IPS)  Pre-  and  Post- ASSET  Training  vs.  Experts 


Anatomy  vs  Procedural  Steps  in  Pre  to  Post  Phase  2  Participants 


-  all_anatomy_avg,all.  techavg  to  all_anatomy_avg,all_tech_avg 

Figure  9a.  Vectors  showing  IPS  change  Pre  to  Post  ASSET  for  the  40  Phase  2  surgeons 


Figure  9b.  Nearest  Neighbor  Pre-  and  Post-ASSET  (n=40)  vs.  Expert  (n=10)  Anatomy  vs.  Technical  Skill 
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Figure  10  (a-c):  Skill  Degradation  with  Time  -  Error  Occurrence  and  Recovery  Since  ASSET 


Figure  10a.  Critical  Errors  vs.  Surgeon  Study  Arm  +/- 1  s.d.  suggesting  a  refresher  ASSET  Course  every  ~  2.5  years  could 

be  beneficial  to  surgeons 
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Figure  10b.  %  Critical  Technical  &  Management  Errors  for  all  vascular  procedures  by  study  arm  with  mean  number  of 

years  since  ASSET  Training 


112 


Critical  Technical  Errors  &  Error  Recovery  Pre- 
and  Post-ASSET  and  mean  1.2  and  2.5  Years  Later 
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Figure  10c.  Demonstrates  the  substantial  decrease  in  Critical  Technical  Errors  after  ASSET  but  the  increase  in  such 

errors  1.2  to  2.5  years  later. 
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Background:  Peripheral  vessel  exposure  and  fasciotomy  are  core  trauma  procedural 
competencies.  Because  of  endovascular  hemorrhage  control,  diminished  vehicle-occupant 
injuries  and  lower  incidence  of  penetrating  trauma,  fewer  open  surgical  procedures  are  needed 
for  trauma.  Advanced  Surgical  Skills  for  Exposure  in  Trauma  (ASSET)  training  is  one  way  to  fill 
this  gap.  We  tested  resident  competence  before,  immediately  after  and  on  re-evaluation  up  to  1.5 
years  following  ASSET  training,  and  compared  resident  performance  after  training  and  re- 
evaluation  to  that  of  experts. 

Methods:  Forty  (PGY  2-6)  residents  from  13  different  training  programs  were  assessed  by 
trained  evaluators  using  a  validated  procedure -based  trauma  readiness  index  (TRI)  checklist 
scoring  system  (including  knowledge,  anatomy,  management,  procedural  steps,  technical 
points),  Global  Ratings  Scale  score  (GRS),  errors  and  time  to  complete  the  procedure1.  The 
residents  were  compared  to  10  expert  traumatologists  (mean  14  years  attending-level  experience 
from  6  different  level  1  trauma  centers).  In  response  to  standardized  scripts,  residents  and  experts 
performed  three  vascular  exposures  (axillary  [AA],  brachial  [BA],  femoral  arteries  [FA])  and  a 
lower  extremity  fasciotomy  [FAS]  in  fresh  cadavers.  Cluster  analysis,  general  linear  modeling 
(GEM),  univariate  and  MANOVA  analysis  were  used  for  TRI,  GRS  and  error  comparisons 
between  residents  and  experts.  A  priori  sample  size  calculation  required  36/40  (90%)  residents  to 
be  followed  up  for  up  to  1.5  years  to  detect  skill  degradation 

Results:  TRI  improved  (P  <0.001)  after  training  in  40  residents.  Specifically,  comparing  post¬ 
training  performance  to  pre-training,  they  demonstrated  a  43%  increase  in  anatomy  knowledge,  a 
57%  increase  in  procedural  steps,  a  decrease  in  frequency  of  errors  from  60%  to  20%,  and 
decreased  time  to  complete  the  procedures  by  2.5  minutes.  Five  residents  did  not  improve  their 
TRIs  after  training.  38/40  (95%)  residents  returned  for  re-evaluation  at  a  mean  of  1.2  years  later. 
There  were  no  differences  in  resident  performance  judged  by  TRI  1.2  years  later  versus 
immediately  post-training.  TRI  in  21/38  (55%)  residents  was  within  one  nearest  neighbor 
classifier  of  the  experts,  who  were  significantly  (p  <0.05)  better  performers  as  a  cohort  judged  by 
TRI  and  GRS.  Number  of  errors  and  error  recovery  rate  were  not  different  between  the  residents 
on  re-evaluation  and  the  expert  cohort.  Members  of  both  expert  and  resident  cohorts  failed  to 
decompress  at  least  one  FAS  compartment,  often  the  anterior  and/or  deep/posterior. 

Conclusion:  Anatomy  and  procedural  steps  were  key  skills  learned  with  ASSET.  The 
performance  of  peripheral  vessel  exposure  by  most  residents  upon  follow-up  was  within  one 
nearest  neighbor  classifier  of  expert  performance.  However,  both  experts  and  residents  made 
errors,  especially  failure  to  decompress  all  FAS  compartments.  In  the  cohort  of  38  residents 
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evaluated,  no  skill  degradation  was  detected  mean  1.2  years  after  ASSET  training.  Five  residents 
did  not  improve  with  training. 

1  J  Surg  Educ  2015;  72:  1278-128 

Supported  by  W81XWH- 13-2-0028 


Evaluating  the  Elemental  Components  of  Surgical  Skill 
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Background:  The  military  needs  a  method  to  assess  trauma  surgical  readiness  for  deployment.  We 
utilized  a  validated  evaluation  tool  to  assess  trauma  surgical  skills  following  the  Advanced  Surgical  Skills 
for  Exposure  in  Trauma  (ASSET)  course. 

Methods:  83  surgeons  were  evaluated  performing  axillary  (AA),  brachial  (BA),  femoral  (FA)  artery 
exposures  and  lower-extremity  fasciotomy  (FAS)  on  unpreserved  cadavers.  Previously  validated 
performance  metrics1,  standardized  script,  two  co-located  evaluators,  and  a  tablet  application  score  sheet 
were  used  for  evaluation  with  checklists  (Trauma  Readiness  Index — TRI),  global  rating  scale  (GRS),  total 
time,  and  errors.  Group  1:  38  PGY  2-6  residents  evaluated  12-18  months  after  ASSET  training;  Group  2: 
35  practicing  surgeons  (pediatric,  plastic,  orthopedic,  general  or  critical  care  surgeons)  evaluated  2-4 
years  after  ASSET  training;  Group  3:  10  experts  (mean  14  years  as  attending  traumatologists).  Kruskal- 
Wallis  test  and  general  linear  modeling  (GLM)  compared  Groups  to  identify  effects  of  months  since 
ASSET  training  and  interval  experience  since  training  including:  numbers  of  trauma  patient  evaluations, 
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numbers  of  upper  extremity  (UE)  and  lower  extremity  (LE)  procedures  determined  by  survey  and  case 
logs  as  high,  medium,  low,  and  relationship  to  components  of  TRI  including:  anatomy,  procedural  steps, 
knowledge  and  overall  GRS  readiness  to  perform. 

Results:  Group  3  surgeons  scored  significantly  higher  in  overall  TRI  and  each  of  the  skill  components 
compared  to  Group  1  and  2,  (p<0.05).  Group  2  TRI  was  less  than  Group  1  in  the  skill  components 
“procedural  steps”  (Group  1  =  71±21,  Group  2  =  64±25,  Group  3  =  80±18  percent,  p<0.05)  and  Group  2 
made  more  “errors”  (Group  1  =  2.4+0.66,  Group  2  =  4.1+1.01,  Group  3  =  2. 2+0.7  errors/surgeon, 
p<0.05),  indicating  that  fewer  procedures  were  correctly  completed  and  errors  increased  with  longer  time 
since  ASSET  training.  Low  interval  experience  with  UE,  FAS  and  number  of  trauma  patients  evaluated 
were  significantly  (p  <0.04)  related  to  TRI.  GLM  showed  interval  experience  of  number  procedures 
performed,  trauma  patients  evaluated,  anatomy,  knowledge,  procedural  steps  were  significant  factors  (p  < 
0.03)  in  Group  2  surgeon  skill  degradation.  Time  since  ASSET  was  a  factor  for  FAS. 

Conclusions:  Group  2  practicing  surgeons  had  more  errors,  lower  TRI  and  were  considered  less  ready  to 
perform  UE  and  FAS  procedures  on  average  28  months  after  ASSET  training.  The  key  factors  associated 
with  interval  skill  degradation  included  number  of  trauma  patients  evaluated  and  number  of  UE  and  FAS 
procedures  performed.  Anatomy,  procedural  steps,  and  knowledge  were  features  of  performance 
evaluated  by  TRI  showing  degradation  in  Group  2  surgeons;  all  such  features  are  amenable  to  re -training. 
In  the  Group  2  cohort  studied,  interval  experience  since  ASSET  training,  not  time,  generally  determined 
skill  degradation,  reduced  readiness  and  increased  procedural  errors. 
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Background:  Surgical  residents  have  increasingly  limited  exposure  to  vascular  trauma 
management  due  to  reduced  on-call  hours,  replacement  of  open  hemorrhage  control  with  non- 
surgical  radiological  balloon  occlusion  and  embolization,  fewer  vascular  trauma  cases 
nationwide,  and  new  blood-use  protocols  reducing  the  need  for  open  surgical  interventions  to 
control  bleeding.  We  previously  showed  residents  express  a  moderately  high  level  of  confidence 
in  their  ability  to  perform  specific  vascular  exposures1.  This  self-reported  confidence  did  not 
match  that  of  co-located  evaluators  who  assessed  the  residents  as  they  performed  procedures 
during  their  evaluations.  This  study  reports  self-confidence  of  a  cohort  evaluated  before  and  after 
ASSET  training  and  12  to  18  months  later.  We  hypothesize  that  residents  will  better  judge  their 
own  skill  after  the  ASSET  course  than  before  when  compared  to  evaluator  performance  ratings, 
resulting  in  self-perceptions  that  more  accurately  reflect  reality. 


Methods:  Forty  PGY2-7  surgical  residents  were  recruited  to  participate  in  a  validation  study  of 
the  Advanced  Surgical  Skills  for  Exposures  in  Trauma  (ASSET)  course.  Each  surgeon 
performed  four  procedures:  axillary  artery,  brachial  artery,  and  femoral  artery  exposure  and 
control,  and  lower  extremity  fasciotomy  on  unpreserved  cadavers  at  three  separate  evaluations. 
These  evaluations  occurred  prior  to  taking  the  ASSET  course  (Pre-evaluation),  within  four 
weeks  after  taking  ASSET  (Post-evaluation),  and  38  out  of  the  40  surgeons  returned  to  be 
evaluated  again  12  to  18  months  later  (mean  ~1.2  years)  (Retention-evaluation).  Before  and  after 
each  evaluation,  the  surgeons  self-assessed  their  baseline  confidence  in  anatomical 
understanding  and  procedure  performance  using  a  5 -point  Likert  scale.  During  the  three 
evaluations,  different  pairs  of  trained  co-located  evaluators  assessed  each  surgeon’s  anatomical 
knowledge  and  surgical  performance  for  each  procedure  using  global  ratings  on  a  5 -point  Likert 
scale.  Each  before  and  after  self-assessment  score  was  then  compared  to  these  corresponding 
global  ratings  using  the  Student  t-test  with  a  set  at  /;<(). 05. 

Results:  For  all  three  evaluations,  residents  consistently  rated  their  understanding  of  anatomy 
(p<0.04)  and  surgical  performance  (p<0.03)  higher  than  evaluators  for  both  the  femoral  artery 
and  fasciotomy  procedures.  The  greatest  difference  occurred  after  the  Pre-Evaluation  for  femoral 
artery  anatomical  understanding  (Surgeon:  3.05  +/-  1.19,  Evaluator:  1.96  +/-  0.76)  and 
fasciotomy  surgical  performance  (Surgeon:  3.55  +/-  0.89,  Evaluator:  2.35  +/-  1.01).  Residents 
rated  their  anatomical  knowledge  higher  (p<0.005)  than  evaluators  for  the  brachial  artery 
procedure  after  their  Post-evaluation  and  before  and  after  their  Retention-evaluations.  Surgical 
performance  was  self-reported  higher  (p<0.03)  than  evaluators  before  and  after  their  brachial 
artery  Post-  and  Retention-evaluations.  The  greatest  difference  for  both  surgical  performance  and 
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anatomical  understanding  of  the  brachial  artery  procedure  occurred  after  the  Retention- 
Evaluation  (Anatomy  Surgeon:  4.11  +/-  0.61,  Evaluator:  2.24  +/-0.80;  Performance  Surgeon: 
3.97  +/-  0.72,  Evaluator:  2.26  +/-  0.81). 

Conclusion:  Residents  overrate  their  anatomical  understanding  and  performance  abilities  even 
after  their  Retention-evaluation  12  to  18  months  later.  Although  evaluator  ratings  increase 
overall  with  ASSET  demonstrating  training  benefits,  resident  surgeon  self-perception  does  not 
reflect  their  trauma  surgical  competency.  Such  operative  skills  are  best  judged  with  evaluations 
performed  by  independent  observers  using  validated  measurements. 
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Background:  Domains  such  as  piloting,  driving  or  laparoscopic  surgery  support  the  collection 
and  analysis  of  motor  control  data  because  human  action  is  executed  through  instruments  with 
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restricted  range  of  motion.  However,  domains  using  instrumentation  with  unlimited  motion, 
such  as  open  surgery,  cannot  exploit  instrument-dependent  data  collection  methods. 
Compensatory  approaches  employing  hand-motion  sensors  risk  task  interference  and 
complicate  performance  analysis.  Standard  observational  judgment  is  costly  and  logistically 
challenging.  We  tested  color  coded  gloves  and  identification  of  surgical  hand-tools,  as  a  proof- 
of-concept  video-based  approach  to  sensor-free  automated  surgeon  performance  evaluation 


Methods:  We  employed  computer  vision  algorithms  (CV)  to  detect  hand  motion  change,  with 
video  and  Shannon  Joint  Entropy  (speed,  acceleration  direction,  (SAD)  change)  analysis  to 
demonstrate  the  potential  for  a  partially  autonomous  evaluation  of  surgical  technical 
performance  evaluation.  Hand-motion  SAD  obtained  by  entropy  was  correlated  with  an 
independently  validated  observational  performance  metric  (Individual  Procedure  Score1  (IPS)). 
We  evaluated  surgical  technical  skill  for  open  surgery  vascular  exposure  and  control  for  trauma 
using  axillary  artery  exposure  and  control  (AA)  among  experts  surgeons,  residents  (both  before 
and  after  training)  and  anatomy  demonstrators,  all  performing  AA  on  fresh  cadavers. 


Results:  Time  to  pectoralis  major,  number  of  instrument  changes,  SAD  and  IPS  for  3  types  of 
operators  differed  among  resident  surgeons,  anatomists  and  experts.  Resident's  SAD  and  IPS 
differed  between  post-training  and  a  skill  retention  evaluation  1  year  after  training.  SAD  data 
showed  changes  convergent  with  those  of  IPS.  SAD  for  2  experts  was  7.29/7.31/3.25,  had  less 
hand  motion  change  than  2  surgical  residents  SAD  of  8.55/8.62/3.25  and  same  resident  SAD  1 
year  later  of  8.47/8.63/3.4  .  Both  expert  and  resident  surgeons  had  lower  SAD  than  2 
anatomists  of  9.15/9.17/3.29.  IPS  were  79%  (experts),  75%/62%  (residents  immediately  after 
and  1  year  after  training).  Anatomist's  IPS  was  not  applicable  as  both  were  trained  IPS 
evaluators.  Procedure  instrument  changes  were  anatomists  45,  experts  23,  residents 
before/after  training  and  1  year  later  were  97/35/50  instrument  changes  respectively  .  Time  (in 
seconds)  to  pectoralis  minor  was  anatomist  99,  experts  64,  residents  before/after  training  and 
1  year  later  842/  100/120.  The  CV  algorithm  is  robust  and  locks  onto  only  hands  with  colored 
gloves  and  is  not  distracted  by  other  colored  surgeon-assistant  gloves  (video  available) 


Conclusions:  These  preliminary  findings  using  surgeon  hand-motion  SAD  have  potential  for  a 
generalizable,  semi-automated,  open  vascular  surgery  performance  evaluation  tool.  Addition  of 
miniature  sensors  on  instruments  and  machine-learning  driven  surgical  gesture  recognition 
(incision,  scissors,  vessel  loop,  suturing  etc)  could  enhance  automation  and  even  avoid  labor- 
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intensive  video  analysis  of  procedural  steps.  Hand  motion  SAD  entropy  from  CV  analyses  were 
congruent  with  the  resource-intensive  IPS  evaluations.  SAD  without  sensors  can  discriminate 
levels  of  surgical  skill,  operators  and  training,  showing  reduction  in  experts  compared  to 
anatomists  and  resident  surgeons. 
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Background:  An  efficient  method  of  trauma  core  procedural  competency  evaluation  is  needed 
to  assess  progress  in  training  and  readiness  for  surgeon  deployment.  Because  on-call  hours  were 
restricted  in  2003  and  endovascular  control  of  hemorrhage  has  reduced  use  of  open  vascular 
control  in  civilian  practice,  open  trauma  surgical  procedural  training  has  declined.  We  assessed 
the  validity  of  a  Trauma  Readiness  Index  (TRI)  performance  metric  for  three  vascular  exposures 
and  lower-extremity  fasciotomy  (FAS)  assesing  TRI  with  blind  video  evaluations  compared  to 
co-located  evaluators.  Such  blind  video  evaluations  would  greatly  simplify  logistics  and  reduce 
inherent  biases  of  assessing  surgeon  performance  of  trauma  core  surgical  competencies. 

Methods:  We  video  recorded  performance  of  axillary  (AA),  brachial  (BA)  and  femoral  artery 
(FA)  vascular  exposure  and  lower  extremity  FAS  on  fresh  cadavers  by  40  PGY  2-6  residents 
from  13  different  training  programs  using  head-mounted  cameras.  Two  co-located  trained 
evaluators  assessed  residents  with  a  standardized  script,  checklists  (TRI),  Global  Rating  Scales 
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(GRS),  time  required  to  complete  the  procedures  and  number  of  errors1,  before  and  after  training. 
One  surgeon  in  each  pre-training  tertile  of  TRI  for  each  of  the  four  procedures  was  randomly 
identified  for  blind  video  review.  The  same  12  surgeons  were  video-recorded  repeating  the  same 
procedures  within  1  month  after  training.  Five  trained  evaluators  independently  reviewed  all  96 
randomly-arranged  blinded  videos.  Inter-rater  reliability/consistency  and  intra-class  correlation 
coefficient  (ICC)  were  compared  by  co-located  versus  video  review  of  TRI  and  errors.  Study 
methodology  and  bias  were  judged  by  Medical  Education  Research  Study  Quality  Instrument 
(MERSQI)  and  the  Quality  Assessment  of  Diagnostic  Accuracy  Studies  (QUADAS-2)  criteria. 

Results:  All  40  residents  returned  and  were  re-evaluated  after  training  performing  the  same 
procedures.  TRI  was  significantly  increased,  time  to  complete  the  procedures  and  errors  were 
reduced  (all  p  <0.001)  with  training.  There  were  no  differences  (p  >0.5)  in  TRI  for  the  12 
surgeons  whose  videos  were  reviewed,  whether  evaluators  were  co-located  or  reviewed  video 
recordings.  Video  evaluator  consistency  was  good  (0.3-0. 8).  Video  and  co-located  evaluators 
were  in  total  agreement  (p  =1.0)  for  error  recognition.  ICC  was  0.73-0.92  for  video-rater 
agreement.  Correlation  of  video  vs  co-located  evaluations  was  0.5-0. 9.  Except  for  BA,  blinded 
video  evaluators  could  discriminate  (p  <  0.002)  between  procedures  performed  pre-  versus  post¬ 
training.  Video  views  of  BA  were  confounded  by  upper  arm  obesity  and  difficulty  obtaining 
adequate  images  to  discriminate  structures  that  could  be  visualized  by  co-located  reviewers. 
Study  methodology  by  MERSQI  criteria  scored  15.5/  18,  QUADAS-2  showed  low  bias  risk  in 
blind  video  review. 

Conclusion:  Evaluation  of  performance  by  video  review  rather  than  requiring  co-located 
evaluators  would  simplify  the  logistics  of  competency  evaluations  and  assessing  surgeon 
readiness  for  deployment.  Video  evaluations  of  trauma  core  procedural  competencies  for  AA, 

FA  and  FAS  with  TRI  are  un-biased,  valid  and  have  potential  for  formative  assessments  of 
competency. 
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Background:  Surgical  residents  have  increasingly  limited  exposure  to  vascular  trauma 
management  due  to  reduced  on-call  hours,  replacement  of  open  hemorrhage  control  with  non- 
surgical  radiological  balloon  occlusion  and  embolization,  fewer  vascular  trauma  cases 
nationwide,  and  new  blood-use  protocols  reducing  the  need  for  open  surgical  interventions  to 
control  bleeding.  We  previously  showed  residents  express  a  moderately  high  level  of  confidence 
in  their  ability  to  perform  specific  vascular  exposures1.  This  self-reported  confidence  did  not 
match  that  of  co-located  evaluators  who  assessed  the  residents  as  they  performed  procedures 
during  their  evaluations.  This  study  reports  self-confidence  of  a  cohort  evaluated  before  and  after 
ASSET  training  and  12  to  18  months  later.  We  hypothesize  that  residents  will  better  judge  their 
own  skill  after  the  ASSET  course  than  before  when  compared  to  evaluator  performance  ratings, 
resulting  in  self-perceptions  that  more  accurately  reflect  reality. 


Methods:  Forty  PGY2-7  surgical  residents  were  recruited  to  participate  in  a  validation  study  of 
the  Advanced  Surgical  Skills  for  Exposures  in  Trauma  (ASSET)  course.  Each  surgeon 
performed  four  procedures:  axillary  artery,  brachial  artery,  and  femoral  artery  exposure  and 
control,  and  lower  extremity  fasciotomy  on  unpreserved  cadavers  at  three  separate  evaluations. 
These  evaluations  occurred  prior  to  taking  the  ASSET  course  (Pre-evaluation),  within  four 
weeks  after  taking  ASSET  (Post-evaluation),  and  38  out  of  the  40  surgeons  returned  to  be 
evaluated  again  12  to  18  months  later  (mean  ~1.2  years)  (Retention-evaluation).  Before  and  after 
each  evaluation,  the  surgeons  self-assessed  their  baseline  confidence  in  anatomical 
understanding  and  procedure  performance  using  a  5 -point  Likert  scale.  During  the  three 
evaluations,  different  pairs  of  trained  co-located  evaluators  assessed  each  surgeon’s  anatomical 
knowledge  and  surgical  performance  for  each  procedure  using  global  ratings  on  a  5-point  Likert 
scale.  Each  before  and  after  self-assessment  score  was  then  compared  to  these  corresponding 
global  ratings  using  the  Student  t-test  with  a  set  at  /;<(). 05. 

Results:  For  all  three  evaluations,  residents  consistently  rated  their  understanding  of  anatomy 
(p<0.04)  and  surgical  performance  (p<0.03)  higher  than  evaluators  for  both  the  femoral  artery 
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and  fasciotomy  procedures.  The  greatest  difference  occurred  after  the  Pre-Evaluation  for  femoral 
artery  anatomical  understanding  (Surgeon:  3.05  +/-  1.19,  Evaluator:  1.96  +/-  0.76)  and 
fasciotomy  surgical  performance  (Surgeon:  3.55  +/-  0.89,  Evaluator:  2.35  +/-  1.01).  Residents 
rated  their  anatomical  knowledge  higher  (p<0.005)  than  evaluators  for  the  brachial  artery 
procedure  after  their  Post-evaluation  and  before  and  after  their  Retention-evaluations.  Surgical 
performance  was  self-reported  higher  (p<0.03)  than  evaluators  before  and  after  their  brachial 
artery  Post-  and  Retention-evaluations.  The  greatest  difference  for  both  surgical  performance  and 
anatomical  understanding  of  the  brachial  artery  procedure  occurred  after  the  Retention- 
Evaluation  (Anatomy  Surgeon:  4.11  +/-  0.61,  Evaluator:  2.24  +/-0.80;  Performance  Surgeon: 

3.97  +/-  0.72,  Evaluator:  2.26  +/-  0.81). 

Conclusion:  Residents  overrate  their  anatomical  understanding  and  performance  abilities  even 
after  their  Retention-evaluation  12  to  18  months  later.  Although  evaluator  ratings  increase 
overall  with  ASSET  demonstrating  training  benefits,  resident  surgeon  self-perception  does  not 
reflect  their  trauma  surgical  competency.  Such  operative  skills  are  best  judged  with  evaluations 
performed  by  independent  observers  using  validated  measurements. 
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Background: 

Limits  in  on-call  hours  have  reduced  operative  experience  for  residents.  The  cadaver-based 
Advanced  Surgical  Skills  for  Exposure  in  Trauma  (ASSET)  course  fills  this  training  gap. 
Cadaver  use  has  limitations  including  cost,  availability,  and  staffing  demands.  Hyper-realistic 
synthetic  models  may  provide  an  alternative  to  cadaver  training.  We  compared  same  surgeon 
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performance  on  synthetic  models  and  cadavers  to  determine  inter-changeability  for  formative 
evaluation. 


Methods: 

75  surgeons  (n  =  40  PGY  2-7  residents  <  4  weeks  after  ASSET;  n  =  35  surgeons  including 
faculty  mean  2.5  years  after  ASSET)  exposed  and  controlled  axillary  (AA),  brachial  (BA),  and 
femoral  arteries  (FA),  and  performed  lower  extremity  fasciotomy  (FAS).  Starting  order  was 
randomized  to  cadavers  or  models  (Operative  Experience  Inc.).  Participants  were  evaluated 
using  individual  procedure  scores  (IPS)  and  aggregate  IPS  for  4  procedures,  a  trauma  readiness 
index  (TRI).  Statistical  analysis  used  student  t-tests.  P-values  <0.05  were  considered  significant. 


Results 

For  the  same  surgeon,  TRI  was  significantly  higher  on  the  model  compared  to  the  cadaver  in 
both  residents  and  faculty  groups  (0.67+0.01  vs  0.64+0.01,  p=0.01;  0.7+0.01  vs  0.66+0.01, 
p<0.01,  respectively).  For  both  residents  and  faculty  average  error  rates  for  all  procedures  were 
lower  for  the  model  (19  vs  29.5,  p<0.01;  12.7  vs  28,  p<0.01,  respectively).  Completion  times 
were  also  lower  on  the  models  (6.2+1.59  vs  9.9+2.47,  p<0.01;  4.9+1.75  vs  9.9+2.82,  p<0.01, 
respectively).  Faculty  group  had  higher  IPS  on  the  model  compared  to  the  cadaver  (AA 
0.67+0.07  vs  0.61±0.1,  p<0.01;  BA  0.79±0.05  vs  0.68±0.12,  p<0.01;  FA  0.65±0.08  vs 
0.59+0.11,  p<0.01;  and  FAS  0.69+0.08  vs  0.64+0.1,  p<0.01).  For  residents  IPS  was  only  higher 
for  the  BA  procedure  (0.75+0.07  vs  0.67+0.09,  p<0.01). 


Conclusion 

Same  surgeons  completed  all  four  procedures  almost  twice  as  quickly  with  fewer  errors  and  a 
higher  TRI  performance  score  on  model  than  cadaver,  indicating  the  relative  ease  of  the  model. 
The  model,  with  easily  discemable  and  standardized  anatomic  structures,  fails  to  capture  the 
complexity  and  variability  of  the  cadaver.  For  residents,  the  models  may  be  useful  in  the  early 
stages  of  training  to  understand  critical  steps  of  complex  procedures.  Residents  with  multiple 
procedural  practices  on  cadavers  before  model  may  account  for  why  the  model-IPS  was  only 
different  on  the  particularly  easy  BA  procedure.  Because  the  same  surgeon  makes  fewer  errors 
in  the  models  than  the  cadavers  and  because  of  higher  TRI  and  ease  compared  to  cadavers, 
models  are  insufficient  to  assess  competency  or  skill  degradation. 
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